Prefix Caching

Prefix caching stores the KV cache from previous requests so a new request that shares a prefix (system prompt, few-shot examples, conversation history) skips the prefill cost for those tokens. vLLM, SGLang, and TGI all support it; llama.cpp added basic support in mid-2024.

For chat with a long system prompt, prefix caching cuts TTFT by 80%+ on every turn after the first. For RAG with a few-shot template, the same template is paid for once per server lifetime instead of once per request.

Cache invalidation is by exact-prefix match — change a single token in the system prompt and the cache misses. Some implementations hash chunks for partial matching.

A customer-support bot runs a 1,500-token system prompt (persona instructions, tool definitions, few-shot examples) ahead of every user turn on a vLLM server. Without prefix caching, each new request re-runs prefill on those 1,500 tokens before it ever touches the user's actual question, adding real latency to every single turn. With prefix caching enabled, the KV cache for that fixed prefix is computed once and reused across every request that matches it exactly — turn two of the same conversation, and turn one of a brand-new conversation with the identical system prompt, both skip straight to processing just the new tokens. The team learns the hard way that prefix caching is exact-match: a developer who templates in today's date at the top of the system prompt unknowingly invalidates the cache on every request across a day boundary, and TTFT quietly regresses until they move the date to the end of the prompt instead.