Sampling, and the knobs providers are taking away

At the end of every forward pass a model holds a list of scores, one per token in its vocabulary, and something has to turn that list into a single next word. That something is sampling. It is a separate step from the model itself, and until recently you could tune it. On some models you now cannot, and this lesson sorts out which ones.

1.A forward pass ends in a distribution, not a word

The last layer of the network produces one number per vocabulary entry, called a logit. A softmax function turns that list into probabilities that add up to one. The model never outputs a word. It outputs a distribution over every word it knows, and a separate rule picks one entry from it.

The simplest rule is greedy decoding: always take the highest-probability token. It sounds obviously right, and it is not. Holtzman and colleagues showed that text chosen purely by likelihood comes out bland and repetitive, a problem they named neural text degeneration. The safest word, taken every step, turns into a loop.

Greedy decoding does not fail by being wrong. It fails by being boring.

So real serving stacks sample. That brings the opposite risk: now and then you draw a near-zero token from the tail, and the answer looks broken. The three knobs below manage that trade.

2.Three knobs on the same distribution

All three act before or during sampling, and all three answer the same question: how much of that probability list is fair game?

Temperature and top-k are the blunter tools. Temperature squeezes the whole distribution by one factor, whether the model is sure or torn. Top-k fixes a headcount with no link to the shape. Top-p fixes that: its cutoff is cumulative probability, so it narrows when the model is confident and widens when it is not.

A worked example makes this concrete. Take one step with five candidate tokens, logits 3, 2, 1, 0 and -1. At temperature 1 the softmax gives about 64, 23, 9, 3 and 1 percent. Divide every logit by 0.5 first and the top token rises to about 87 percent; divide by 2 and it falls to about 43. Top-p at 0.9 then keeps three tokens, two, or four. On a confident step (logits 6, 2, 1, 0, -1) the same p keeps one, while a top-k of 3 would keep three on both.

None of the three has anything to do with whether the answer is correct. They govern variety and fluency. A wrong answer at a low temperature is still wrong; it is just wrong more consistently.

3.Some knobs are now locked, and what to use instead

As of Oct 2026, Anthropic's model-deprecations page lists temperature, top_p, and top_k as deprecated on Claude 4.7 and later models. On those models a non-default value returns a 400 error, and the page says to leave the parameters out and steer with prompting. The migration guide for the current Sonnet model agrees, and adds that Sonnet 4.6 and earlier, and Haiku 4.5, still accept them. Neither page gives a reason or states the default value, so omit the argument rather than assuming a number.

Other providers are less clear. OpenAI's reasoning guide and Google's thinking guide, as of Oct 2026, do not mention temperature at all. One preprint, What Aggregate Scores Miss, says three OpenAI models take no temperature setting, but that is a single source, unconfirmed against an OpenAI page. Read each model's parameter list, and keep sampling settings in a per-model table, not hard-coded into the call. Self-hosted engines are different again: as of Oct 2026, Hugging Face's text-generation-inference still lists temperature scaling, top-p, and top-k among its features, though its docs now call it maintenance mode and point to vLLM, SGLang, llama.cpp and MLX. The Running a model yourself lesson covers that route.

The failure you will meet first is a ported script. Code written for an older model passes temperature=0.2. Point it at a current id and it breaks, in one of two ways. The Python SDK from version 1.0 on has removed the three arguments from its request type, so the call raises a TypeError before anything is sent; I saw exactly that on version 1.15.0 in Oct 2026. An SDK or a raw HTTP call that still accepts the argument gets the 400 instead. The fix is the same: delete the argument.

The quieter failure is code that leaned on temperature 0 to mean "same input, same output", such as a test that compares two replies. That was never promised. Floating-point arithmetic on a GPU is not strictly associative, and your request is batched with other people's, so the numbers can differ slightly between calls. Infrastructure can be at fault too. Anthropic's September 2025 postmortem describes two separate bugs. A 16-bit versus 32-bit precision mismatch, found in December 2024, could drop the highest-probability token at temperature 0. A different approximate top-k step, exposed in August 2025, returned wrong results for some batch sizes. That is one vendor's account of its own bugs, but it shows how many layers sit between "temperature 0" and the token you get. The Timeouts, fallbacks, and failures that look like success lesson tells the rest of that story.

What to do instead depends on what you wanted. For consistency, catch variation after the call: validate the output against a schema, and treat a case that flips between runs as a sign the prompt is ambiguous, not as noise to suppress. For accuracy on a task with one right answer, use self-consistency: run the same prompt several times and keep the answer that recurs. Wang and colleagues reported clear gains this way on arithmetic and commonsense benchmarks, with models from 2022.

When voting does not apply. It needs short answers you can compare, such as a label or a number; open prose gives you five essays and no majority. It costs about five times the tokens of one call. It cannot fix an error the model makes every time, because the same wrong answer wins the vote. And models that already reason before answering may gain less from it; whether that holds for your model and task is untested here, so measure it before you pay for it.

from collections import Counter
from anthropic import Anthropic, APIStatusError

# Check the models page for a current id; as of Oct 2026 this one is active.
MODEL = "claude-sonnet-5-5"

client = Anthropic()

# Older tutorials say lower temperature for consistency. On this model it is not yours to set.
try:
    client.messages.create(
        model=MODEL,
        max_tokens=50,
        temperature=0.2,
        messages=[{"role": "user", "content": "Say hello."}],
    )
except (TypeError, APIStatusError) as err:
    # Python SDK 1.x: TypeError, before any request. An SDK that still accepts it: a 400.
    print(type(err).__name__, "-", err)

def ask(question: str) -> str:
    resp = client.messages.create(
        model=MODEL,
        max_tokens=4096,                     # covers any thinking plus the answer
        output_config={"effort": "low"},
        messages=[{"role": "user", "content": question + " Reply with the number only."}],
    )
    # Read the reply by block type: a thinking block may come first.
    return next(b.text for b in resp.content if b.type == "text").strip()

def self_consistent_answer(question: str, tries: int = 5) -> str:
    answers = [ask(question) for _ in range(tries)]
    return Counter(answers).most_common(1)[0][0]   # the answer that showed up most often

print(self_consistent_answer(
    "A shop sells eggs only in boxes of 6 and 7. What is the largest number of eggs you cannot buy exactly?"
))

Run it with an ANTHROPIC_API_KEY set. The first call prints TypeError - Messages.create() got an unexpected keyword argument 'temperature' on the current Python SDK, or a 400 message on an SDK that still accepts the argument. The second prints a number; the right answer is 29. The voting half was not run against the live API when this was written, so run it before you trust it.