Quantization

Quantization is the process of reducing a model's numeric precision to shrink its memory footprint with minimal quality loss. A 70B-parameter model in FP16 takes 140 GB; in Q4_K_M (a 4-bit GGUF format) it takes about 40 GB and runs on a single RTX 4090.

Common formats for local inference: GGUF (the llama.cpp format, with K-quants like Q4_K_M, Q5_K_M, Q8_0), EXL2 (NVIDIA-only, used by ExLlamaV2), AWQ and GPTQ (NVIDIA-only, post-training methods), and MLX (Apple Silicon).

Important non-obvious detail: Q4_K_M isn't really 4 bits — it uses 6-bit precision on attention and feed-forward layers and 4-bit elsewhere, averaging about 4.83 bits/parameter. This is why naive "params × 4 / 8" sizing under-predicts actual file size by ~20%.

An operator with a single RTX 4090 (24GB VRAM) wants to run Llama 3.3 70B locally. At FP16 that's 140GB — impossible. Downloading the Q4_K_M quantization instead brings it to roughly 40GB, which still won't fit on one card, but Q4_K_M of a 32B model (around 20GB) fits comfortably with room left for KV cache. This is the recurring sizing exercise in local AI: pick the quant level that trades quality for footprint until the math closes against your actual VRAM. Dropping from Q8_0 to Q4_K_M roughly halves memory with a small, usually acceptable quality hit for most chat and coding tasks; going lower (Q3 or Q2) starts introducing noticeably degraded outputs, especially on smaller base models that have less redundancy to lose. Always check /will-it-run against the actual GGUF file size, not a back-of-envelope params-to-bits estimate.