AWQ

AWQ (Activation-aware Weight Quantization) is a 4-bit quantization method designed for fast inference on NVIDIA GPUs. It's the production-default quant for vLLM and SGLang serving. AWQ analyzes activation distributions during calibration to identify "salient" weight channels and protects them at higher precision while aggressively quantizing the rest. Result: ~2% quality loss vs FP16 on most reasoning benchmarks; ~3.5× memory savings.

Operator notes that matter: AWQ is NVIDIA-only (no AMD, no Apple). It requires a calibration dataset (default ones ship with the AutoAWQ library — usually fine). vLLM 0.7+ ships AWQ kernels with full PagedAttention compatibility; throughput on A100/H100 is within 5% of FP16 at much lower VRAM cost. Compared to GPTQ: AWQ is generally faster at inference; GPTQ has more aggressive quant variants. Compared to GGUF Q4_K_M: AWQ is faster on serving runtimes; GGUF works on more backends but lacks the kernel-level vLLM optimization.

When to use AWQ: production NVIDIA serving with vLLM/SGLang, where throughput-per-VRAM-dollar matters. When NOT to use AWQ: AMD or Apple deployments (use GGUF Q4_K_M instead), or workloads where you need the absolute strongest quant quality at all costs (use FP8 if you have H100).

An operator serving Qwen 2.5 32B on a single A100 80GB for a production chat API needs to fit the model, KV cache, and headroom for concurrent requests. FP16 alone takes ~64GB, leaving little room to batch. Switching to AWQ 4-bit brings the weights down to ~18GB, freeing over 40GB for KV cache — enough to serve dozens of concurrent 8K-context sessions instead of two or three. They deploy it on vLLM with --quantization awq, and because vLLM's AWQ kernels integrate with PagedAttention, throughput stays close to FP16 despite the memory savings. The catch surfaces during evaluation: on a coding-heavy eval suite, AWQ's ~2% quality dip shows up as slightly more syntax errors in edge cases, so they keep an FP16 fallback path for the highest-stakes completions rather than trusting AWQ everywhere blindly.