INT4

INT4 is a quantization format that stores each model weight using 4 bits, reducing memory usage by roughly 4× compared to FP16 (16-bit) and 8× compared to FP32 (32-bit). In practice, INT4 quantization is applied to transformer-based large language models (LLMs) to fit larger models into limited VRAM. For example, a 70B parameter model that requires ~140 GB in FP16 can be compressed to ~35 GB in INT4, making it runnable on a single 48 GB GPU. The trade-off is a small drop in output quality (perplexity increase of ~0.5–1.0) compared to higher-bit formats.

INT4 quantization works by mapping the original floating-point weight values to a small set of 16 possible integer values (since 4 bits = 16 levels). The mapping is typically done per group of weights (e.g., 32 or 128 weights) using a scaling factor and zero-point, a technique called group-wise quantization. Common INT4 quantization methods include GPTQ (post-training quantization), AWQ (adaptive weight quantization), and GGML/GGUF's Q4_0, Q4_1, Q4_K_M variants. The choice of method affects both the quality and the inference speed. INT4 models run fastest on hardware with dedicated integer matrix multiplication units, such as NVIDIA GPUs with Tensor Cores (compute capability 7.0+) and Apple M-series chips with ANE. On older GPUs without efficient INT4 support, the runtime may dequantize weights on the fly, reducing throughput.

A 7B parameter model like Llama 2 7B in FP16 requires 14 GB of VRAM. Using INT4 quantization (e.g., Q4_K_M in GGUF format), the same model fits in ~4 GB, allowing it to run on an RTX 3060 (12 GB) with room for a 4K context window. On an RTX 4090 (24 GB), INT4 enables running a 34B model like Yi-34B at Q4_K_M (20 GB) comfortably.

In llama.cpp, operators select INT4 by choosing a GGUF file with a 'Q4' variant (e.g., 'llama-2-7b.Q4_K_M.gguf'). When using Ollama, pulling a model like 'llama3.1:8b' automatically downloads a Q4_0 quantized version. In LM Studio, the model browser shows VRAM requirements for each quantization; selecting a Q4 variant reduces the bar from red (insufficient VRAM) to green. In vLLM, INT4 can be enabled via the --quantization awq flag when serving an AWQ-quantized model.