GPU Memory
Last Updated: July 29, 2026 | By Mihail Sebastian | AI Dictionary
Dedicated high-bandwidth memory on a graphics card holding model weights, activations, and data batches – usually the first resource training runs out of.
What is GPU Memory?
GPU memory is the dedicated high-bandwidth memory on a graphics card that holds the data the GPU computes on: model weights, activations, gradients, and input batches.
It is separate from the system RAM the CPU uses, and in deep learning it is the resource that runs out first. How much a card carries sets a hard ceiling on the size of model you train and the batch size you feed it.
How GPU Memory Works
GPU memory sits on the card itself, next to the processor, with far higher bandwidth than system RAM. That closeness is what lets thousands of CUDA threads read and write data fast enough to stay busy.
During training, several things compete for the same fixed space. The model’s weights live there permanently; the optimizer keeps its own bookkeeping values per weight; every forward pass stores activations that the backward pass needs; and each data batch occupies room while it is processed.
When the total exceeds the card’s capacity, the job stops with an out-of-memory error. The standard responses are smaller batches, mixed-precision arithmetic that stores numbers in half the bytes, quantization, or recomputing activations instead of storing them.
Example of GPU Memory
A team fine-tunes a 7-billion-parameter language model on a card with 24 GB of memory. Stored at 16 bits, the weights alone take about 14 GB – over half the card before training begins.
Then the rest arrives. The Adam optimizer keeps two extra values per parameter, and activations pile up with every layer of every batch. The first training step fails with an out-of-memory error.
The team responds step by step: cut the batch size, switch the optimizer state to lower precision, and recompute activations during the backward pass rather than storing them. The job now fits, at the cost of somewhat slower steps. Every large-model training run involves this same budgeting exercise.
Related AI terms: CUDA · TPU · Memory Leak · Quantization · Model Training
Did you like the GPU Memory gist?
Learn about 250+ need-to-know artificial intelligence terms in the AI Dictionary.
Mihail Sebastian — Writes about AI governance, regulation, and the technology behind them. Placeholder bio — replace with a real credential line. About