LLM Sampling With Numbers: Temperature, Top-k, Top-p, Min-p

The model's last layer produces a vector of logits, one real number per token in the vocabulary. Logits are unbounded and only their differences matter. Softmax turns them into probabilities: raise e to the power of each logit, then divide by the sum so the results add up to 1.

Take a model completing "The cat sat on the" with a five-token vocabulary and these logits:

• mat 3.0, sofa 2.0, floor 1.5, roof 0.5, moon -1.0

Softmax gives mat 0.591, sofa 0.217, floor 0.132, roof 0.049 and moon 0.011. A difference of one logit is a factor of e (about 2.7) in probability: mat is 1.0 above sofa and about 2.7 times as likely.

Sampling means drawing a token at random with those probabilities. Drawing 100,000 times from this distribution with the code below gave mat 59.2%, sofa 21.6%, floor 13.3%, roof 4.9% and moon 1.1%, matching the probabilities to within 0.1 point. Every sampler setting in this guide changes those probabilities before the draw.

Here is the implementation used for every number in this guide. It works on a dictionary from token to logit or probability, which is slow but easy to read:

1import math, random2 3def softmax(logits, t=1.0):4    m = max(logits.values())5    e = {k: math.exp((v - m) / t) for k, v in logits.items()}6    z = sum(e.values())7    return {k: x / z for k, x in e.items()}8 9def renorm(p):10    z = sum(p.values())11    return {k: v / z for k, v in p.items()}12 13def top_k(p, k):14    keep = sorted(p, key=p.get, reverse=True)[:k]15    return renorm({t: p[t] for t in keep})16 17def top_p(p, threshold):18    keep, total = [], 0.019    for t in sorted(p, key=p.get, reverse=True):20        keep.append(t)21        total += p[t]22        if total >= threshold:23            break24    return renorm({t: p[t] for t in keep})25 26def min_p(p, base):27    cutoff = base * max(p.values())28    return renorm({t: v for t, v in p.items() if v >= cutoff})29 30def sample(p, rng=random):31    r, acc = rng.random(), 0.032    for t, v in p.items():33        acc += v34        if r < acc:35            return t36    return t

Each truncation function keeps a subset of tokens and calls renorm, which divides by the kept total so the survivors add up to 1 again. That step matters: removing tokens does not just delete their probability, it gives it to the tokens that remain, in proportion to their size.

Production engines do the same operations on tensors: they sort or partially sort the logits once, apply a mask, and sample on the GPU. The maths is identical.

Temperature divides every logit by T before softmax. Below 1 it stretches the gaps between logits, so the top token takes more of the probability; above 1 it shrinks the gaps, so the distribution flattens. For the five-token example:

• T = 0.5: mat 0.839, sofa 0.114, floor 0.042, roof 0.006, moon 0.000
• T = 1.0: mat 0.591, sofa 0.217, floor 0.132, roof 0.049, moon 0.011
• T = 2.0: mat 0.400, sofa 0.243, floor 0.189, roof 0.115, moon 0.054

Halving the temperature doubles the logit gaps. mat and sofa go from one logit apart to two, a factor of e² (about 7.4) in probability. Doubling it does the reverse: moon becomes five times more likely than at T = 1. Temperature never changes the ranking; mat stays first at every setting. As T approaches 0, all the probability goes to the top token, which is why engines treat T = 0 as greedy decoding rather than dividing by zero.

The trade-off is diversity against coherence. Holtzman et al. found in open-ended generation with GPT-2 Large that temperatures below 0.9 severely increased repetition. Very high temperatures produced text whose perplexity was far above human text: the model kept drawing unlikely tokens and then had to continue from them. Temperature alone cannot fix this, because it scales the whole tail up or down together.

Top-k keeps the k most likely tokens and discards the rest. With k = 3 on the five-token example, roof and moon go, and the survivors renormalize to mat 0.629, sofa 0.231 and floor 0.140. The 6% that roof and moon had is shared out among the three.

The weakness is that k is fixed while the distribution's shape is not. Holtzman et al. illustrate both failure modes. Consider two contexts:

• Peaked: "The capital of France is" gives Paris 0.91, Lyon 0.04, a 0.03, the 0.015, Nice 0.005. Top-3 keeps Lyon and a, and renormalization raises their combined chance from 7% to 7.2%. One draw in fourteen goes off course.
• Flat: "My favourite food is" spreads probability over ten reasonable options from pizza 0.16 down to soup 0.04 and the 0.02. Top-3 keeps pizza, pasta and sushi and removes seven reasonable answers.

A small k makes flat contexts bland; a large k lets junk through in peaked ones. No single value fits both, and a long generation passes through both kinds of context many times.

Holtzman et al. proposed nucleus sampling: keep the smallest set of most likely tokens whose probabilities add up to at least p, then renormalize. The pool size now follows the distribution:

• Peaked, p = 0.9: Paris alone has 0.91, which already reaches 0.9, so the pool is one token.
• Flat, p = 0.9: the running total reaches 0.88 after seven tokens and 0.96 after eight, so the pool is eight tokens, dropping only soup and the.

The paper's key claim is that a language model's distribution has an unreliable tail: many low-probability tokens that are individually unlikely but together get drawn often enough to derail long text. Top-p cuts that tail by mass rather than by count. In the paper's GPT-2 Large comparison, nucleus sampling at p = 0.95 produced text with perplexity 13.13 against 12.38 for human text, and 0.36% repetition against 0.28%. Top-k = 40 gave 6.88 and 0.78%, and greedy decoding gave 1.50 and 73.66% repetition. It also had the best combined human and statistical score (HUSE) among the methods compared.

The boundary is that top-p measures mass, not quality. When the distribution is flat because temperature is high, the tail holds most of the mass, and a mass threshold must include it, as the next section shows.

Min-p, proposed by Nguyen et al. (ICLR 2025), keeps every token whose probability is at least p_base × p_max, where p_max is the top token's probability. With p_base = 0.1:

• Peaked: the cutoff is 0.1 × 0.91 = 0.091, so only Paris survives.
• Flat: the cutoff is 0.1 × 0.16 = 0.016, so all ten tokens survive, including the at 0.02.

The flat case shows that min-p is not stricter than top-p in general; here it keeps two more tokens. Its claimed advantage is at high temperature. Take five good tokens (logits 6.0 down to 4.0) and 1,000 junk tokens at logit -2.0, with temperature applied first. At T = 1, junk holds 12.6% of the probability; top-p 0.9 keeps 210 tokens and draws junk 2.9% of the time, while min-p 0.1 keeps only the five good tokens. At T = 2, junk holds 85%; top-p must include 883 junk tokens to reach 0.9 and draws junk 83.4% of the time, while min-p still keeps five. Min-p's cutoff is a ratio to the top token, and the junk tokens stay below a tenth of it until about T = 3.5. By T = 4 both methods draw junk about 97% of the time.

Whether that mechanism produces better text is disputed. The original paper reports gains on GPQA, GSM8K and AlpacaEval creative writing, and in human evaluations. A 2025 re-analysis by Schaeffer et al. concluded that the evidence does not support those claims. It reports that the human evaluations omitted data and used incorrect statistical tests, and that benchmark gains disappear when hyperparameters are tuned equally for each method. Treat min-p as a reasonable truncation rule, not a proven improvement.

When you set temperature and a truncation together, the result depends on which runs first, and engines differ:

• Hugging Face Transformers applies temperature, then top-k, top-p and min-p, in that order.
• llama.cpp's server applies its truncation samplers first and temperature last by default (top_k, then top_p, then min_p, then temperature, with some other samplers in between). Its samplers option changes the order.

With the five-token example at T = 2 and top-p 0.9:

• Temperature first: the flattened distribution needs four tokens to reach 0.9, giving mat 0.423, sofa 0.256, floor 0.200, roof 0.121.
• Temperature last: top-p on the original distribution keeps three tokens (0.940), and temperature then flattens just those: mat 0.481, sofa 0.292, floor 0.227. roof can never be drawn.

With temperature last, truncation decides which tokens are possible and temperature only reshapes the survivors, so a high temperature cannot pull the tail back in. That is also why the high-temperature results for min-p depend on the engine. The same settings copied from one engine to another can produce different output, so record the engine and its sampler order alongside the values.

The trend in hosted APIs is to take these settings away. Claude's Messages API marks temperature, top_p and top_k as deprecated: models released after Claude Opus 4.6 accept temperature only at 1.0 and top_p only at 0.99 or above, for backwards compatibility, and reject any top_k. OpenAI's migration guide for its current models says to remove temperature and top_p when reasoning effort is anything other than none.

The reason is not stated in either document, but the effect is clear: on those models, decoding is the provider's decision. Code that sets temperature=0 for an extraction job will fail on a newer Claude model, and the fix is to remove the parameter, not to find another value.

You keep full control on open-weight models served with vLLM, Transformers or llama.cpp, and on hosted models that still accept the parameters. On models that do not, steer behaviour with other tools: clearer instructions, structured outputs for fixed formats, reasoning-effort settings where offered, and multiple samples when you need variety.

Where you can set the sampler, start from what the task needs and change one setting at a time:

• Extraction and classification: you want the model's single most likely answer. Use greedy decoding or a low temperature, and enforce the format with structured outputs or a schema. Randomness adds nothing here.
• Code, one attempt: a low temperature gives the model's preferred solution. If you can run tests, sampling several candidates at a moderate temperature and keeping one that passes can do better than a single greedy answer, because the candidates differ.
• Chat and explanation: the provider's default (usually temperature 1.0 with light or no truncation) is tuned for this. Change it only with evidence.
• Creative writing and brainstorming: temperature around 1.0 with top-p 0.9–0.95, or min-p 0.05–0.1 on engines that support it. These are the community values the min-p paper evaluated. Raising temperature further is where truncation matters most.

Pick one truncation method. Stacking top-k, top-p and min-p makes the result depend on the engine's order, and the min-p authors report renormalization problems when several are combined.

Then measure. Hold the prompt fixed, generate 20 to 50 outputs per setting, and score them on what the task needs: accuracy for extraction, tests passed for code, a rubric or human preference for writing. Sampler defaults are guesses about your task; a small evaluation replaces the guess. For how to set that up, see Agent Evals and Observability.

Work through these with the sampling.py code above:

1. Compute softmax of the five-token example at T = 0.5, 1.0 and 2.0 and check it against the temperature section. 2. Draw 100,000 tokens at T = 1.0 and confirm each empirical frequency is within 0.5 points of its probability. 3. Apply top_k(p, 3), top_p(p, 0.9) and min_p(p, 0.1) to the peaked and flat distributions, and count the survivors. 4. Build the 1,005-token example (five good tokens with logits 6, 5.5, 5, 4.5 and 4, plus 1,000 junk tokens at -2) and compute the chance of drawing junk under top-p 0.9 and min-p 0.1 at T = 1, 2 and 4. 5. Repeat step 4 with temperature applied after truncation, and explain why min-p never admits junk in that order. 6. If you have an open-weight model in Transformers or vLLM, generate 20 continuations of one prompt at T = 1.5 with top-p 0.9, then with min-p 0.1, and read them side by side.

Pass criterion: your numbers match this guide to three decimal places for steps 1, 3 and 4, and you can explain from step 5 why the same settings behave differently in two engines.