INT4
Definition pending
We've cataloged "INT4" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
INT4 quantization compresses model weights to 4 bits (0.5 bytes) per parameter. A 7B model goes from 14 GB (FP16) to ~4 GB (Q4_K_M). Quality loss is ~1–3% on benchmarks for well-quantized models. This is the format that makes local AI possible on consumer hardware — without INT4, you'd need 48 GB+ GPUs for 7B models.
Workflow example
GGUF INT4 variants: Q4_0 (fastest, slightly lower quality), Q4_K_M (balanced, recommended default), Q4_K_S (smaller, lower quality). For most use cases, Q4_K_M is the right choice. Test: load the same prompt with Q4_K_M and Q5_K_M, compare outputs. If quality difference is imperceptible, save the 20% memory. Use Q4 for chat, Q5–Q8 for tasks requiring precision (code, math).