Throughput

Throughput measures how much work a system completes per unit time — typically tokens-per-second across all concurrent requests. Distinct from latency, which measures a single request's time.

A vLLM server with continuous batching can serve dozens of concurrent users with 5-10× the aggregate throughput of a single-stream llama.cpp setup, because batching amortizes the cost of reading model weights from VRAM across multiple requests' tokens.

For solo local use you mostly care about latency, not throughput. For self-hosted multi-user deployments (a team using a shared local LLM) throughput is the key metric. The right runner choice differs: ExLlamaV2 wins single-user; vLLM wins multi-user.

A small team runs a shared coding assistant on a single RTX 6000 Ada (48 GB) serving 15 engineers. Deployed on Ollama (single-stream), the server queues requests — engineer #10's request waits for #1-9 to finish, and effective throughput collapses even though the GPU sits idle between requests. Switching to vLLM with continuous batching lets the same GPU interleave all 15 requests' decode steps into shared forward passes, keeping the GPU busy and multiplying aggregate tok/s. The individual engineer's latency per request barely changes, but total system throughput scales because weight-loading cost is amortized across concurrent streams. This is the core reason self-hosted team deployments standardize on vLLM or SGLang instead of the single-user llama.cpp servers that work fine for a solo laptop setup.