LLM Temperature and Sampling Parameters

Temperature is the most adjusted and least understood setting in the API. The folk explanation — that it is a creativity dial, and that turning it down makes the model more accurate — is wrong in a way that leads people to make real mistakes.

Here is what it actually does.

What the model produces

At each step, the model does not choose a word. It produces a probability distribution over its entire vocabulary — tens of thousands of tokens, each with a score. Something has to turn that distribution into one token. That something is the decoding strategy, and temperature, top-p and top-k are its parameters.

Crucially, the model is unchanged by any of them. Its beliefs about what comes next are fixed the moment the forward pass completes. These settings only govern how you sample from those beliefs.

Temperature

Temperature reshapes the distribution before sampling.

Low temperature sharpens it. Probable tokens become more probable, unlikely ones less so. At 0 it collapses to always taking the single most likely token — greedy decoding.

High temperature flattens it. The gap between likely and unlikely narrows, so rarer tokens get a real chance.

So the honest description is: temperature controls how willing the model is to pick a token it considers less likely. Not creativity, not accuracy, not confidence.

Why the accuracy framing misleads: at temperature 0 the model still produces its most likely token, and if its most likely token is a fabricated citation, you now get that fabrication deterministically, every time. Lowering the temperature makes output more consistent, which people experience as more reliable. It does not make it more true. See why AI models hallucinate.

One more thing worth knowing: temperature 0 is not a guarantee of identical output. Batching, hardware non-determinism and mixture-of-experts routing all introduce variation. Treat it as as deterministic as this system offers, not as a reproducibility guarantee.

Top-p, and why it exists

Top-p — nucleus sampling — takes a different approach. Instead of reshaping the whole distribution, it truncates it: sort tokens by probability, keep adding until their cumulative probability reaches p, discard everything else, sample from what remains.

The elegance is that the cut adapts. Where the model is confident, a handful of tokens reach 0.9 and the pool is small. Where it is genuinely uncertain, the pool is large. The threshold responds to the shape of the distribution rather than imposing a fixed size on it.

This came from Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi's The Curious Case of Neural Text Degeneration (ICLR 2020), which established two things that still hold:

decoding strategies alone can dramatically effect the quality of machine text, even when generated from exactly the same neural language model

and that maximisation-based methods — greedy decoding and beam search — produce text that is bland and strangely repetitive, despite likelihood being the training objective. Their proposal, nucleus sampling, allows for diversity while effectively truncating the less reliable tail of the distribution.

That tail is the thing to understand. The bottom of the distribution is a very long list of tokens each holding a tiny probability, which collectively hold a meaningful share. Raise temperature without truncation and you are sampling from that garbage. This is why output at very high temperature degenerates into incoherence rather than becoming interestingly creative — and why top-p exists as a companion.

Top-k is the blunter ancestor: keep the k most likely tokens regardless of their probabilities. Simpler, and worse in exactly the case top-p handles — a fixed k is too permissive when the model is confident and too restrictive when it is not.

What to actually set

Change one at a time. Most providers apply temperature and top-p together, and adjusting both makes the effect of either impossible to reason about. Pick one — top-p if available — and leave the other at its default.

For extraction, classification, structured output and tool calls: temperature 0 or very close. You want the same input to produce the same output, and there is no upside to variety when the correct answer is a specific value. Pair it with a schema — getting reliable structured output from LLMs.

For code: low, but not always zero. Zero is right for a targeted edit. If the first attempt is wrong, a slightly higher temperature is a cheap way to get a genuinely different attempt rather than the same wrong one again.

For ordinary prose and analysis: leave the defaults alone. Provider defaults are tuned for general use and are a reasonable starting point far more often than people assume.

For brainstorming and variation: raise it, moderately. The useful range is smaller than people expect. Going very high does not produce more creative output; it produces less coherent output. If you want genuinely diverse ideas, the better lever is usually the prompt — asking for approaches from named, different angles — rather than the sampler.

When you need N different options, generate N times rather than asking for ten in one response. A single response listing ten ideas is one sample, and items three through ten are conditioned on items one and two, so they converge. Separate calls are independent samples and genuinely more diverse.

The larger point

Sampling parameters are a small lever. Reaching for temperature is often a sign that the prompt is underspecified — inconsistent output usually means the instructions permit the variation you are seeing, and the fix is to constrain the task rather than the sampler.

Get the prompt right first: the anatomy of a great prompt, system prompts vs user prompts, and few-shot vs zero-shot. Three examples of the output shape you want will do more for consistency than any temperature setting.

And whatever you change, measure it. These settings interact with your specific prompt and task in ways no general guidance predicts, which is what how to evaluate and test your prompts is for. A/B a parameter change against a real eval set, or you are tuning on vibes.

One caution on reasoning models: several providers restrict or ignore these parameters for reasoning-mode models, where the sampling regime is managed internally. Check the documentation for the model you are calling rather than assuming — and see how to prompt reasoning models.

Sources

• Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes and Yejin Choi, The Curious Case of Neural Text Degeneration, ICLR 2020 — introduced nucleus (top-p) sampling; decoding strategy alone dramatically affects output quality, maximisation-based decoding produces bland and repetitive text, and truncating the unreliable tail is what makes higher-diversity sampling usable