Deterministic Decoding

Deterministic decoding means same prompt → same output, every time. Achieved by setting temperature to 0 (always pick the highest-probability token) and pinning the random seed for any tiebreaks.

Sounds simple, isn't. Even at temperature 0, GPU floating-point non-associativity can produce different logits across runs (especially with batch-size variation), which can flip ties. True bit-exact reproducibility requires single-batch, deterministic kernels (CUBLAS_DETERMINISTIC, cuDNN deterministic mode), and pinned seed everywhere.

For local AI evaluation, "deterministic enough" usually means temperature 0 + single batch + same hardware/runtime version. Cross-runtime reproducibility (llama.cpp ↔ vLLM) is essentially never bit-exact even with identical sampling settings.

An engineer debugging a quantization regression runs the same prompt through an FP16 baseline and a Q4_K_M GGUF build, both at temperature 0, expecting identical completions modulo the quant error — but the FP16 outputs differ slightly between two runs on the same GPU. The cause: batch size changed between runs (1 vs 4 concurrent requests), and GPU kernels use non-associative floating-point reduction, so summation order shifts logits by fractions of a bit, occasionally flipping a near-tied argmax. Pinning batch size to 1 and enabling CUBLAS_DETERMINISTIC (or the llama.cpp single-thread path) restores run-to-run identical output. This matters for regression testing: comparing a new quantization format against baseline requires knowing whether observed diffs are real quality loss or just non-deterministic noise — bit-exact reproducibility is a precondition for that comparison to mean anything.