KV Cache

The KV cache stores the key and value tensors from previous attention computations so the model doesn't recompute them at every generated token. Without it, generation speed would be O(n²); with it, each new token is roughly O(n).

The catch: KV cache memory scales linearly with context length. The formula is 2 × num_layers × num_kv_heads × head_dim × context_length × bytes_per_element. For Llama 3.3 70B at FP16, every 1K tokens of context costs about 320 MB of VRAM.

Modern models use Grouped-Query Attention (GQA), where num_kv_heads << num_attention_heads, dramatically reducing cache size. Llama 3.1 8B has 32 attention heads but only 8 KV heads — a 4× cache reduction over old MHA architectures. Quantized KV cache (FP8 or INT4) halves or quarters this further.

An operator running Llama 3.1 8B locally for long-document summarization notices VRAM climbing steadily as context grows, even though the model weights themselves are fixed at around 5GB in Q4. That's the KV cache growing with every token of context fed in. Thanks to GQA's 4x reduction in KV heads versus old multi-head attention, an 8B model can sustain a much longer context on a single consumer GPU than a same-size model built on legacy MHA would allow. When the operator pushes toward the model's max context window and starts hitting out-of-memory errors, the fix isn't necessarily a bigger GPU — quantizing the KV cache to FP8 (supported in llama.cpp and vLLM) often buys back enough headroom to finish the job without touching the model weights themselves.