Time to first token (TTFT)

TTFT (time-to-first-token) is the latency between sending a prompt and receiving the first generated token. It's dominated by the prefill phase — the model processing the input prompt before generation begins. For a 1K-token prompt on RTX 4090 + Llama 3.1 8B, TTFT is typically 50-150 ms; for a 32K prompt, it can rise to 1-3 seconds.

Why TTFT matters operationally: chat UX feels broken at TTFT > 1 second; agent loops with frequent short tool-call turns are dominated by TTFT, not decode tok/s. A runtime that wins on decode but loses on prefill (e.g. some llama.cpp configurations) feels unresponsive in agentic workloads even when the steady-state tok/s is competitive.

Optimization levers: PagedAttention (vLLM, SGLang) eliminates prefill recomputation of cached prompts. Speculative decoding can compress TTFT for short outputs (controversial — speculative is more decode-side). Prefix caching (RadixAttention in SGLang) is the single biggest TTFT win for agent workloads where the system prompt is stable across requests. Flash Attention 2/3 reduces prefill compute meaningfully on long-context queries. Quantization choice affects TTFT differently than decode — AWQ-INT4 has slightly slower prefill kernels than FP16 on some models because of dequantization overhead.

An operator building an agentic coding tool notices users complain the assistant "feels laggy" even though decode throughput benchmarks at a respectable 70 tok/s on their RTX 4090. Profiling reveals the real problem: each tool-call turn re-sends a growing system prompt plus tool schemas, and with a 6K-token context the prefill phase alone takes almost 900ms before generation even starts — and this happens dozens of times per agent session. Switching from a stock llama.cpp server to SGLang with RadixAttention prefix caching means the repeated system-prompt prefix is cached across requests instead of recomputed every turn, cutting effective TTFT to under 100ms for subsequent calls in the same session. Decode tok/s barely changes, but the agent loop suddenly feels responsive — a reminder that for short, frequent turns, TTFT dominates perceived speed far more than steady-state tokens per second.