10. PagedAttention

Memory fragmentation destroys inference efficiency. During autoregressive generation, KV cache entries grow dynamically as new tokens generate. Pre-allocating memory for maximum context length wastes VRAM—when generating 100 tokens, the memory reserved for 4096 tokens sits idle.

PagedAttention solves this by managing KV cache as virtual memory pages, similar to operating system memory management. Memory allocates on-demand in 4KB (or configurable) pages, eliminating internal fragmentation.

# vLLM PagedAttention configuration
from vllm import LLM, SamplingParams

llm = LLM(
    model="meta-llama/Llama-2-70b-hf",
    tensor_parallel_size=2,           # Distribute across GPUs
    gpu_memory_utilization=0.90,      # Use 90% of available VRAM
    max_num_seqs=256,                 # Maximum concurrent sequences
    max_num_batched_tokens=8192,     # Batching budget
    block_size=16,                    # KV cache page size
)

# Streaming with natural preemption
sampling_params = SamplingParams(
    temperature=0.8,
    top_p=0.95,
    max_tokens=512,
)

outputs = llm.generate(
    ["Explain quantum entanglement", "Write a haiku about AI"] * 128,
    sampling_params
)

Block management enables critical optimizations:

Prefix caching: Identical system prompts across requests share KV cache pages. In a chat application where every request begins with the same system message, this saves 10-30% computation.

Automatic batching: Requests arriving mid-generation automatically batch with active requests, maximizing GPU utilization.

KV cache eviction: When memory pressure requires eviction, PagedAttention selects blocks with lowest future utility based on sequence position.

# Prefix caching example
system_prompt = "You are a helpful assistant. Always be concise."

# All requests share system_prompt KV cache
requests = [
    {"prompt": system_prompt + "What is Python?"},
    {"prompt": system_prompt + "Explain recursion"},
    {"prompt": system_prompt + "What are closures?"},
]

# First request caches prefix
# Subsequent requests reuse cached prefix
# ~15-25% speedup from prefix sharing

Memory allocation strategy affects performance. Block size of 16 balances fragmentation against allocation overhead. Smaller blocks (4, 8) improve memory utilization but increase metadata overhead. Larger blocks (32, 64) reduce overhead but increase fragmentation.

# Block size comparison
configs = [
    {"block_size": 4, "best_for": "Many short sequences"},
    {"block_size": 16, "best_for": "General purpose"},
    {"block_size": 32, "best_for": "Long sequences, fewer sequences"},
]

for config in configs:
    llm = LLM(
        model="meta-llama/Llama-2-70b-hf",
        block_size=config["block_size"],
    )
    # Benchmark with your expected workload

Failure modes: Block size mismatch between serving engine and model configuration causes runtime errors. Verify that block_size aligns with model's attention implementation.