EXL2

EXL2 is the ExLlamaV2 quantization format. NVIDIA-only, single-stream-throughput-optimized. Allows fractional bit-rates (e.g. 4.65 bits per weight) by mixing higher-precision weights for "important" channels with lower-precision for the rest. Used through ExLlamaV2 + TabbyAPI as the OpenAI-compatible serving wrapper.

The EXL2 case operationally: the fastest single-stream tok/s on consumer NVIDIA for most models. Beats vLLM AWQ by 10-25% on per-stream throughput at the same VRAM target. Compatibility cost: EXL2 doesn't support continuous batching the way vLLM does, so multi-tenant concurrency is weaker. EXL2 doesn't run on AMD or Apple.

When to use EXL2: solo-user or small-team consumer-NVIDIA deployments where peak per-stream tok/s matters more than concurrency. The dual-3090 NVLink + ExLlamaV2 combo is one of the highest-throughput single-stream setups under $2,000. When NOT to use EXL2: production serving with multi-user concurrency (use vLLM/SGLang with AWQ instead), AMD/Apple targets, or workloads where you need the broader quant ecosystem (GGUF for portability).

A solo developer running a coding assistant on a single RTX 4090 wants maximum tokens/sec for one interactive session rather than serving multiple users. They quantize a 32B model to EXL2 at 4.65 bits per weight — a fractional rate flat GPTQ/AWQ can't express — via ExLlamaV2's calibration pass, then serve it through TabbyAPI with an OpenAI-compatible endpoint. Single-stream throughput comes in meaningfully ahead of an equivalent AWQ model served through vLLM on the same card, confirming EXL2's single-stream-optimized design. When they later need to serve the same model to five concurrent teammates, EXL2's lack of continuous batching becomes the bottleneck — each additional stream roughly divides throughput rather than sharing GPU work efficiently — so they migrate that deployment to vLLM with AWQ despite the lower per-stream ceiling, keeping gguf as the portable fallback for anyone without an NVIDIA card.