Decode (Token Generation)

Decode is the second phase of LLM inference: generating one output token at a time, autoregressively. Each decode step does a small matrix-vector multiplication against the full model weights, then samples from the output distribution.

Decode is memory-bandwidth-bound, not compute-bound. The throughput ceiling is model_size / memory_bandwidth. A 7B model in Q4 (5 GB) on an RTX 4090 (1008 GB/s) tops out at ~200 tok/s in theory, with real numbers typically 60–75% of that.

This is why decode tok/s scales with VRAM bandwidth (HBM > GDDR7 > GDDR6X > unified memory) far more than with FLOPS. Batch size and speculative decoding are the main levers to push decode past the bandwidth wall.

An operator benchmarking a 7B Q4_K_M model (about 5 GB) on an RTX 4090 (1008 GB/s bandwidth) expects near the theoretical ~200 tok/s ceiling but measures only around 120-140 tok/s. That's within the normal 60-75% real-world efficiency band for decode, so nothing's actually wrong — the gap comes from attention overhead, kernel launch cost, and sampling, not a misconfiguration. When they later test the same model on a laptop with LPDDR5 unified memory at roughly 120 GB/s, decode speed drops proportionally, confirming the memory-bandwidth-bound relationship rather than a compute limit. To push past the wall without new hardware, they enable speculative decoding with a small draft model, which lets the GPU verify several tokens per memory pass instead of one.