VRAM (Video RAM)

VRAM is the dedicated memory on a GPU. For local AI, VRAM capacity is the single most important spec — it determines which models you can load. The relationship between model size, quantization, and VRAM is the central calculation behind every "will this run" question.

Rules of thumb: a model in FP16 needs about 2GB per billion parameters; in Q4 about 0.6 GB per billion. Add 15-30% overhead for KV cache, activation memory, and runtime buffers. A 7B model in Q4 fits comfortably in 8 GB VRAM; 70B Q4 needs 48 GB; 70B FP16 needs 140 GB.

Important: VRAM is gated, not just slow. If a model spills into system RAM via CPU offload, generation drops from 40 tok/s to 2-3 tok/s — a usability cliff. Apple Silicon's unified memory bypasses this distinction, treating all RAM as VRAM, which is why M-series Macs punch above their weight for local LLMs.

An operator with a 12GB RTX 3060 wants to run Llama 3.1 8B and does the math: FP16 needs about 16GB (2GB/B × 8B), already over budget before accounting for KV cache. Dropping to Q4 quantization brings weights down to roughly 4.8GB (0.6GB/B × 8B), leaving comfortable headroom for context and activations within the 12GB ceiling. The same operator later tries a 13B model at Q4 (~7.8GB) with a long 16K context and hits the VRAM ceiling mid-generation — llama.cpp's automatic CPU offload kicks in, and throughput craters from a usable 35 tok/s to an unusable 3 tok/s. Checking /will-it-run before downloading a new model avoids this cliff entirely.