Request Batching

Request batching packs multiple inference requests into a single forward pass to amortize the cost of loading model weights from VRAM. Since decode is memory-bandwidth-bound, doubling the batch size roughly doubles aggregate tok/s without slowing per-request latency much, until the batch saturates compute.

Static batching (Ollama, llama.cpp default) waits for a fixed number of requests before launching. Continuous batching (vLLM, TGI) joins requests mid-flight. Dynamic batching (TensorRT-LLM) adapts batch size to load.

For single-user local AI, batching is invisible. For multi-user serving, it's the difference between 1 and 50 concurrent users on the same hardware.

An operator benchmarking a shared vLLM deployment sees aggregate throughput climb steadily as they raise max_num_seqs from 1 to 32 — decode is memory-bandwidth-bound, so each added concurrent request rides along almost free until the GPU's compute finally saturates around request 40 or so, where throughput growth flattens and per-request latency starts climbing noticeably. This is request batching in action: continuous batching lets vLLM insert new requests into an in-flight batch instead of waiting for a fixed group to fill, unlike Ollama's simpler static queueing. For their solo laptop llama.cpp setup at home, this entire mechanism is invisible — one user, one request at a time, no batching benefit to chase. Batching only matters once you're serving more than one concurrent user.