Throughput vs Latency

Throughput is aggregate tokens generated per second across all in-flight requests; latency is wall-clock time for a single request (TTFT + total decode time). Optimizing for one trades against the other.

Bigger batch sizes increase throughput but raise per-request latency, because each forward pass takes longer when more requests are packed in. Smaller batches do the opposite.

Local single-user AI almost always cares about latency: TTFT under 500 ms, decode at "feels-instant" speeds (>15 tok/s for chat, >40 for code). Multi-user serving cares about throughput: tokens per dollar per hour. Don't pick a config from a throughput benchmark when you're optimizing latency, or vice versa.

An operator tunes a vLLM deployment for a coding assistant used by one developer and mistakenly copies batch-size settings from a benchmark tuned for a 50-user support bot. Latency immediately suffers: TTFT creeps past a second and per-token decode slows, because the large batch size that maximizes aggregate throughput also means each individual forward pass is doing more work, delaying every single request in flight. For their actual single-user case they want the opposite — batch size near 1, KV cache tuned for one long context rather than many short ones, and TTFT as the metric they watch. The fix is recognizing which side of throughput vs latency their workload lives on before copying someone else's benchmark config: solo local chat and coding tools optimize for latency; shared team deployments optimize for throughput.