Temperature (sampling)

Temperature is a sampling parameter that controls the randomness of token selection during text generation. It scales the logits (raw scores) before applying softmax: lower values (e.g., 0.1) make the model more deterministic by favoring high-probability tokens, while higher values (e.g., 1.5) increase diversity by flattening the probability distribution. At temperature 0, the model always picks the most likely token (greedy decoding). Operators tune temperature to balance coherence and creativity; typical ranges are 0.6–0.9 for chat and 0.7–1.2 for creative writing.

Temperature is applied by dividing each logit by the temperature value before the softmax function. A temperature of 1 leaves logits unchanged; values below 1 sharpen the distribution (making high-probability tokens even more likely), while values above 1 flatten it (giving lower-probability tokens a better chance). This is distinct from top-k or top-p sampling, which truncate the candidate set. Temperature interacts with other samplers: for example, using high temperature with top-p can still produce coherent output if the nucleus is small. In practice, temperature is set per-request or per-model; some runtimes (like llama.cpp) allow dynamic adjustment during generation.

When running Llama 3.1 8B via Ollama for a Q&A task, setting temperature to 0.2 yields factual, repetitive answers. For a story generation, raising it to 0.9 produces more varied plot directions. On an RTX 4090, both settings run at the same ~100 tok/s because temperature only affects sampling, not inference speed.

In llama.cpp, temperature is set via --temp 0.8 on the command line or in the server API as "temperature": 0.8. In Ollama, you can set it in the Modelfile with PARAMETER temperature 0.7 or override per request: curl http://localhost:11434/api/generate -d '{"model":"llama3.1","prompt":"...","options":{"temperature":0.5}}'. In LM Studio, it's a slider in the UI under 'Sampling Parameters'.