Sampling (Decoding)

Sampling is the process of converting model logits into output tokens. Common strategies: greedy (temperature 0), random sampling (temperature > 0), top-k, top-p (nucleus), min-p, typical sampling, mirostat. Most runtimes let you stack them — top-p over top-k over temperature.

The sampling configuration has more impact on perceived quality than most users assume — temperature 0.1 vs 0.7 vs 1.2 produces output that feels like different models. Defaults vary widely: Ollama defaults to temperature 0.8, vLLM to 1.0, llama.cpp to 0.8.

For evaluation, document the full sampling config when reporting numbers. "Llama 3.1 8B got 70 on MMLU" is meaningless without specifying whether that's at temperature 0 or with sampling.

Two operators compare the same model — both running Llama 3.1 8B via different runtimes — and get noticeably different personalities: one feels terse and repetitive, the other verbose and occasionally incoherent. The cause turns out to be defaults: Ollama's temperature=0.8 versus a custom vLLM deployment left at temperature=1.0 with no top-p cap, compounded by one setup also applying a repeat penalty the other doesn't. Neither is "wrong," but neither is a fair comparison of model quality either. When benchmarking a new quantization format against a baseline, the fix is to pin every sampling parameter (temperature, top-p, top-k, repeat penalty, seed) identically across both runs and log the full config alongside any reported score — otherwise a claimed quality regression might just be a sampling-config mismatch.