Inference

Inference is the act of running a trained model to generate predictions, as opposed to training which produces the model. For LLMs, inference has two distinct phases:

Prefill processes the entire prompt in one parallel pass; this is compute-bound (matrix-matrix operations). Time-to-first-token (TTFT) is the latency from request to first generated token, dominated by prefill.

Decode generates one token at a time autoregressively; this is memory-bandwidth-bound (matrix-vector operations on the full model weights). Tokens-per-second is the headline metric, scaling linearly with hardware memory bandwidth.

For local AI on consumer hardware, decode dominates real-world experience. An RTX 5080 with 960 GB/s bandwidth running an 8B Q4 model (5 GB) achieves a theoretical peak of ~190 tok/s, with real measurements typically 60-75% of that.

An operator benchmarking a new RTX 5080 (960 GB/s memory bandwidth) for local inference loads an 8B model at Q4 (roughly 5GB) and measures two very different numbers. Time-to-first-token on a 2,000-token prompt is dominated by prefill, which is compute-bound and scales with GPU FLOPS — fast on modern silicon since it's a parallel matrix-matrix pass. But sustained generation speed is a decode problem: every token requires streaming the full model weights through memory again, so throughput is bounded by bandwidth divided by model size, not by the GPU's raw compute. In practice the operator sees generation land well below the theoretical bandwidth-derived ceiling once real-world overhead (kernel launch, sampling, attention over a growing KV cache) is accounted for — a gap worth budgeting for when promising a client a specific tokens-per-second SLA.