Prefill (Prompt Processing)

Prefill is the first phase of LLM inference: the model processes the entire prompt in a single parallel pass, building up the KV cache for every prompt token. Prefill is compute-bound — large matrix-matrix operations that saturate tensor cores.

Prefill latency dominates time-to-first-token (TTFT). For a 7B model on an RTX 4090, prefill runs at roughly 3,000–8,000 tokens/sec depending on batch geometry, so a 2K-token prompt takes 250–700 ms before generation even starts.

Optimizations: chunked prefill (process the prompt in slices to overlap with decode), prefix caching (reuse KV from a previous prompt with the same prefix), and Flash Attention (reduce memory traffic during attention).

An operator debugging a slow chatbot response notices the delay before the first token appears is inconsistent — fast for short questions, sluggish for anything with a long system prompt or pasted document. Profiling the request shows the lag tracks prompt length almost linearly: a 200-token question returns in well under 100ms, but a 6K-token document dump takes several seconds before generation starts. That's prefill time, not decode — the model is doing one large compute-bound pass over the whole prompt to populate the KV cache. The fix isn't a faster GPU necessarily; it's prefix caching, so a system prompt that's identical across requests doesn't get reprocessed every time, and chunked prefill in servers like vLLM so a huge prompt doesn't block other users' TTFT on a shared server.