PagedAttention

PagedAttention is the memory layout introduced by vLLM that stores the KV cache in fixed-size blocks (pages), like virtual memory. Each request keeps a page table mapping logical positions to physical blocks; blocks can be shared across requests with identical prefixes.

The win: eliminates the internal fragmentation that wastes 60–80% of KV cache memory in naive contiguous allocations. PagedAttention lets vLLM pack 2–4× more concurrent requests into the same VRAM.

A side benefit: trivial prefix caching — multiple requests sharing a system prompt point at the same physical block, so the prefill cost is paid once.

A team serving a shared local LLM behind vLLM sees a naive server choke: with 32 concurrent chat sessions each holding a growing KV cache in a contiguous block, memory fragments badly, and VRAM runs out well before the GPU's true capacity is reached, capping concurrency far lower than the hardware should allow. Switching to a runtime that implements PagedAttention — vLLM's default — fixes this by allocating KV cache in fixed-size pages instead of one contiguous block per request, eliminating that fragmentation and roughly doubling or better the number of concurrent sessions the same 48 GB card can hold. A bonus shows up immediately: all 32 sessions share the same system prompt, so their KV cache pages for that shared prefix physically overlap, meaning prefill for the system prompt is computed once instead of 32 times.