Q3_K_M Quantization

Q3_K_M is a 3-bit GGUF K-quant averaging ~3.9 bits per parameter. It's the smallest format that still produces usable output for most models.

Quality drops noticeably: perplexity is typically 0.5–1.0 points above FP16, and complex tasks (multi-step reasoning, code) show measurable degradation. For 7B–13B models, Q3_K_M is rarely worth it — drop to a smaller model at Q4_K_M instead. For 70B+ models on consumer hardware, Q3_K_M is the only path that fits in 36 GB or under.

If output starts producing word salad, the model is past its quant cliff; try Q4_K_S or Q4_K_M instead.

An operator has a single 24 GB RTX 4090 and wants to run a 70B model locally rather than downgrade to a smaller one. At Q4_K_M, a 70B model needs roughly 42 GB — nowhere close to fitting. Dropping to Q3_K_M brings it down to around 34 GB, still too large, so they either offload layers to CPU RAM (accepting a heavy tok/s penalty) or accept partial GPU offload. On a 36 GB unified-memory Mac, though, Q3_K_M 70B fits entirely in VRAM-equivalent memory and runs at usable speed. They test it against a coding task first, since that's where 3-bit degradation shows up fastest — if the model starts producing syntactically broken code or forgetting earlier context, that's the quant cliff, and the fix is either a smaller model at Q4_K_M or accepting the quality hit for the larger parameter count.