Latency

Latency measures how fast you get a response. Two metrics matter for local LLMs:

Time to First Token (TTFT) — wall-clock from request to first generated token. Dominated by prefill (compute-bound). Long prompts make TTFT worse linearly. On a 4090, a 1K-token prompt has ~50ms TTFT; a 32K prompt has 1-2 seconds.

Inter-Token Latency — time between consecutive tokens during generation. Inverse of tokens-per-second. Dominated by memory bandwidth in the decode phase.

Distinct from throughput, which measures total tokens-per-second across batched/concurrent requests. A serving system optimized for throughput (vLLM with continuous batching) often has worse single-request latency than a system optimized for latency (ExLlamaV2).

An operator building a coding assistant on an RTX 4090 notices autocomplete feels sluggish only when working in a large file. Profiling shows TTFT balloons from ~50ms with a 1K-token prompt to over a second once the open file plus surrounding context pushes the prompt past 20K tokens — that's prefill cost, not generation speed. Switching to a serving stack with prompt caching (reusing the KV cache across keystrokes instead of reprocessing the whole file each time) collapses TTFT back down, while inter-token latency stays roughly constant since it's bottlenecked by the 4090's ~1008 GB/s memory bandwidth regardless of prompt length. This is why coding tools prioritize prefix caching over raw throughput tuning.