Continuous Batching

Continuous batching (sometimes "iteration-level scheduling") is a serving optimization where new requests join the active batch as soon as one slot finishes, instead of waiting for the whole batch to complete. Pioneered by Orca and now standard in vLLM, TGI, and SGLang.

Compared to static batching, continuous batching delivers 2–10× higher throughput on real workloads where prompts have varying lengths. For local single-user setups, the win is small; the point is keeping a server busy under multi-user load.

Implementation requires per-request KV cache slots and per-iteration scheduling. Not all serving stacks support it — Ollama and llama.cpp's default server are static-batch.

An operator running a small internal API for a team of 15 engineers notices that Ollama's default server chokes when three people query simultaneously — request four waits for request one to fully finish before its slot opens. Switching the backend to vLLM with continuous batching enabled, new requests slot into the running batch the moment any sequence finishes, rather than waiting for the whole cohort to complete. Aggregate throughput on the mixed-length prompts jumps noticeably, since the GPU stays busy instead of idling on the batch's longest sequence. For their own single-user local chat setup on the same laptop, though, they don't bother — continuous batching only pays off once concurrent requests are actually queuing, and paged attention does more of the heavy lifting for memory efficiency in that regime.