Quantization

Quantization is the process of reducing a model's numeric precision to shrink its memory footprint with minimal quality loss. A 70B-parameter model in FP16 takes 140 GB; in Q4_K_M (a 4-bit GGUF format) it takes about 40 GB and runs on a single RTX 4090.

Common formats for local inference: GGUF (the llama.cpp format, with K-quants like Q4_K_M, Q5_K_M, Q8_0), EXL2 (NVIDIA-only, used by ExLlamaV2), AWQ and GPTQ (NVIDIA-only, post-training methods), and MLX (Apple Silicon).

Important non-obvious detail: Q4_K_M isn't really 4 bits — it uses 6-bit precision on attention and feed-forward layers and 4-bit elsewhere, averaging about 4.83 bits/parameter. This is why naive "params × 4 / 8" sizing under-predicts actual file size by ~20%.

An operator with a single RTX 4090 (24 GB VRAM) wants to run Llama 3.1 70B locally. At FP16 the model needs 140 GB — impossible. At Q4_K_M quantization it needs roughly 40 GB, still too large for one card, but Q4_K_M on the 8B variant needs only about 5 GB, leaving 19 GB free for a large KV cache and long context. For the 70B model specifically, they'd instead reach for a 2-bit or 3-bit GGUF quant (accepting more quality loss) or split layers across CPU RAM and GPU VRAM with --n-gpu-layers in llama.cpp. When sizing a purchase, they use /will-it-run rather than hand-computing "params × 4 / 8," since that naive formula underestimates real GGUF file size by roughly 20% thanks to mixed-precision K-quants.