Q2_K Quantization

Q2_K is 2-bit GGUF quantization averaging ~3.0 bits per parameter (with mandatory 4-bit scales and importance metadata). It exists for very large models on very small hardware.

Quality is materially worse than Q4 — perplexity often 1.5–3 points above FP16, with coherence breakdowns on long generations. For most local AI, Q2_K is not recommended; pick a smaller model at higher precision.

The legitimate use case: running a 70B+ model on 24 GB of VRAM, where Q2_K is the only fit. Even then, expect noticeable hallucination and weak instruction-following compared to the same model at Q4.

An operator wants to try a 70B-class model on a single 24 GB GPU without any CPU offload. At Q3_K_M the model still doesn't fit; only Q2_K, at roughly 3 bits per parameter, brings the footprint down to around 26-27 GB of weights — still tight, but close enough that dropping context length or using a smaller KV cache gets it inside 24 GB. Running it, they notice the model starts strong on simple factual queries but drifts into repetition or invents details on longer, multi-turn conversations — classic Q2_K coherence breakdown. The practical call: use it only for quick exploratory queries where a wrong answer is cheap to catch, and treat any output requiring precision (code, citations, math) as unreliable until re-run through a smaller model at Q4_K_M instead.