Sampling, and the knobs providers are taking away

Every model, at the very end of a forward pass, does the same odd thing. It takes everything it has read and turns it into a list of numbers — one per token in its vocabulary — and then something has to turn that list into an actual next word. That something is sampling, and it is a genuinely separate piece of the pipeline from the model itself. It used to be a piece you could tune. Increasingly, it is not.

1A forward pass ends in a distribution, not a word

The last layer of the network produces one number per vocabulary entry, called a logit. A softmax function turns that list into probabilities that add up to one. Nowhere in this does the model output a word — it outputs a distribution over every word it knows, and a separate decision procedure has to pick one entry from it.

The simplest procedure is greedy decoding: always take the single highest-probability token. It sounds like the obviously correct choice, and it is not. Text produced this way is duller and more repetitive than text sampled from the distribution properly, a finding formalised by Holtzman and colleagues under the name "neural text degeneration." Greedy decoding chases the single safest word at every step, and the safest word, repeated, is a loop.

Greedy decoding does not fail by being wrong. It fails by being boring — the same safe phrase, over and over, because the top token is nearly always the safe one.

So real serving stacks sample from the distribution instead of always taking the top entry. That reintroduces a different problem: sampling from the full distribution means occasionally drawing one of the thousands of nearly-zero-probability tokens in the tail, which produces the kind of nonsense word that makes an answer look broken. The three knobs in the next section exist to manage that trade-off — enough randomness to avoid the loop, not so much that the tail gets drawn from.

2Three knobs on the same distribution

All three act before or during sampling, and all three answer a version of the same question: how much of that probability list is fair game?

Temperature and top-k are the older, blunter tools. Temperature rescales the whole distribution by the same amount regardless of shape — a model that is extremely sure of the next token and a model that is genuinely torn between five options both get squeezed by the identical factor. Top-k fixes a headcount that has no relationship to how peaked or flat the distribution actually is; k = 40 throws away nothing useful when the model is confident, and lets in forty options when it should really only be choosing between three.

Top-p was built to fix exactly that. Because its cutoff is defined by cumulative probability rather than a fixed count or a fixed rescaling, it automatically narrows when the model is confident and widens when it is not. That is why many serving stacks favour it over the older knobs: it responds to what the model itself is telling you about its own uncertainty at that specific step, instead of applying the same treatment everywhere.

None of the three, and this is the part worth sitting with, has anything to do with whether the answer is correct. They govern variety and fluency, not accuracy. A wrong answer sampled at a low temperature is still wrong — it is just wrong the same way every time.

3Why the labs are removing the knobs, and what replaces them

Here is the part that has changed recently enough that most write-ups on this topic have not caught up. Anthropic's models released after Claude Opus 4.6 accept exactly one value for temperature — 1.0, kept only for backward compatibility — and reject every other value with a 400. top_p and top_k are restricted the same way. When extended thinking is switched on, the API enforces this directly: thinking requires the default temperature, full stop. OpenAI made the equivalent move on its reasoning line. The knob most tutorials treat as universal is being taken out of your hands on exactly the models built to reason carefully.

The reasoning is not arbitrary. A model that generates a long chain of intermediate steps before answering has been tuned, internally, for how that exploration should behave. A caller flattening or sharpening the distribution mid-chain — usually while trying to make the final answer more consistent — can quietly wreck the reasoning that produces it. Rather than let people do that to themselves by habit, the labs removed the option.

It costs you less than it looks like, because the thing you were probably using it for — a guarantee of identical output — was never real. Temperature 0 has always been an approximation of determinism, not a guarantee of it. Floating-point arithmetic on a GPU is not strictly associative, and production serving batches your request together with other people's, so identical inputs can still land in slightly different numerical company from one call to the next. If your system depended on temperature 0 producing the same string forever, it was already depending on something no provider ever promised.

So what do you reach for instead? For consistency, stop trying to suppress variation at the sampler and catch it downstream: validate the output against a schema, and treat a case that flips between two runs as a sign that prompt is genuinely ambiguous rather than noise to filter away. For accuracy on a reasoning-shaped task, use self-consistency: run the same prompt several times and keep the answer that shows up most often. It recovers much of what a cooler temperature used to buy on tasks with one right answer, without touching a knob that, on the model you are calling, may simply no longer exist.

from anthropic import Anthropic, APIStatusError
from collections import Counter

client = Anthropic()

# Older tutorials say lower this for consistency. On current models it is not yours to set.
try:
    client.messages.create(
        model="claude-sonnet-5",
        max_tokens=10,
        temperature=0.2,
        messages=[{"role": "user", "content": "Say hello."}],
    )
except APIStatusError as err:
    print(err.status_code, "-", err.message)   # 400 - temperature is deprecated for this model

# The replacement lever for consistency is not a smaller number. It is more samples.
def self_consistent_answer(question: str, tries: int = 5) -> str:
    answers = []
    for _ in range(tries):
        resp = client.messages.create(
            model="claude-sonnet-5",
            max_tokens=20,
            messages=[{"role": "user", "content": question}],
        )
        answers.append(resp.content[0].text.strip())
    return Counter(answers).most_common(1)[0][0]   # the answer that showed up most often

print(self_consistent_answer(
    "A shop sells eggs in boxes of 6 and 9. What is the largest number of eggs you cannot buy exactly?"
))

The first call demonstrates the wall you will hit the first time you port an old script to a current model. The second is the actual fix: five cheap calls and a Counter, buying back the reliability a locked sampler no longer will.