Q8_0 Quantization

Q8_0 is llama.cpp's simplest 8-bit GGUF quantization: weights in INT8, one FP16 scale per 32-element block, no zero-point. Each parameter takes about 8.5 bits including the scale.

Q8_0 is the "near-lossless" tier — perplexity is typically within 0.01 of FP16 on standard benchmarks. The cost is size: a 7B model is ~7.6 GB and a 70B is ~75 GB, only ~46% smaller than FP16. For most local-AI hardware, Q8_0 is overkill; Q5_K_M or Q4_K_M deliver 95%+ of the quality at half the memory.

When to actually pick Q8_0: when you're benchmarking quant impact and need a tight upper bound, or when running a model that's already close to your VRAM ceiling and you need every drop of fidelity.

An operator is deciding whether a quality regression they're seeing in a 7B model's outputs is caused by the model itself or by their quantization choice. They download the same checkpoint in both Q4_K_M and Q8_0, run both against a fixed eval set, and compare. Q8_0 comes in at ~7.6GB versus Q4_K_M's ~4.1GB — nearly double the size for a model that already fits comfortably in a 24GB RTX 4090 — but since perplexity is within 0.01 of full FP16, it serves as a reliable near-lossless reference point. When Q4_K_M's outputs diverge meaningfully from Q8_0's on the eval set, that gap is attributable to quantization rather than the underlying model, telling them to bump the deployed quant up to Q5_K_M rather than debugging a phantom model issue.