Why models hallucinate

1There is no "I don't know" token

Every answer a model gives, right or wrong, comes out of the same machine: a forward pass ends in a distribution over the next token, and something gets sampled. Sampling, and the knobs providers are taking away covered the mechanics of that step. What matters here is what is missing from it — there is no separate check the model runs first, no internal dial it reads that says "I am 40% sure" before deciding whether to answer or demur.

Refusing to answer is not a special mode sitting outside generation. It's a sequence of tokens — "I'm not certain," "I don't have reliable information on that" — and those tokens have to win the same competition every other continuation wins: highest probability, given everything the model has seen. A refusal only comes out if it happens to score higher than a specific, confident, wrong answer. Most of the time, for most models, it doesn't.

Think about why. A confident wrong answer and a confident right answer are produced by the identical process — the model has learned that certain question shapes are followed by certain answer shapes, and it produces the shape. "What year was X founded" is reliably followed by a year. If the model has seen the real year during training, it produces that year. If it hasn't, or has seen conflicting years, it still produces a year, because the shape of the continuation — a confident four-digit number — was never conditioned on whether the model actually knew it.

A wrong answer and a right one come off the same production line. The model isn't lying and it isn't guessing in a way it could tell you about — it's doing the one thing it does, which is continue the text.

This is the part worth sitting with before the fix-shaped thinking kicks in: there is nothing to patch here, because nothing is broken. The model is doing exactly what it was built to do. The question is why that produces confident nonsense as often as it does.

2Training rewards a good guess over an honest shrug

Kalai, Nachum, Vempala, and Zhang, in a 2025 OpenAI paper, gave this an actual mechanism instead of a shrug of their own. Their argument has two parts, and both matter.

Part one is about pretraining. A model learns the statistics of its training text. Some facts appear often and consistently — the model converges on them reliably. Other facts appear once, or inconsistently, or not at all — a person's exact birthdate, an obscure paper's page count, a small company's founding year. For that second category, there is nothing in the data that reliably picks out the right token over a plausible wrong one. The paper shows this is close to an unavoidable consequence of training on next-token prediction over data that contains facts stated once: the generation error rate on a fact is bounded below by roughly double the error rate you'd get if you were just classifying "is this specific claim true or false." Generation is a strictly harder problem than verification, and no amount of training scale removes that gap for facts that are sparse in the data.

Part two is the sharper one, and it's about evaluation, not pretraining. Once a model exists, it gets graded — on benchmarks, on human preference ratings, on the metrics a lab optimizes against. Nearly all of those grading schemes work like a multiple-choice exam with no penalty for a wrong guess: a right answer scores full marks, a wrong answer and an admitted "I don't know" score the same zero. Under that scoring, guessing is never worse than abstaining, and it's sometimes better. A model — or a training process shaping a model's behavior toward higher eval scores — that has any nonzero chance of guessing right does strictly better on average by always guessing than by ever admitting doubt.

Put the two parts together: pretraining guarantees the model will sometimes not really know a fact, and the standard evaluation regime guarantees that "not really knowing" gets rendered as a confident guess rather than a flagged uncertainty, because a confident guess is the higher-scoring move under how these systems get graded. Hallucination, on this account, isn't a glitch that slipped through testing. It's what the testing rewards.

import anthropic
from collections import Counter

client = anthropic.Anthropic()

# A fact obscure enough that the model may not have seen it reliably.
question = "What was the exact attendance figure at the 1987 IEEE conference on parallel computing?"

answers = []
for _ in range(5):
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=60,
        messages=[{"role": "user", "content": question}],
    )
    answers.append(resp.content[0].text.strip())

for a in answers:
    print(a)

print("\nDistinct answers:", len(set(answers)))
# A stable, made-up number across all five runs would look identical in tone
# to a genuinely known fact repeated five times — fluency tells you nothing.
# Disagreement across runs is one of the only external signals you get.

Run something like this on a fact you can independently check, and you'll usually see one of two shapes: the same confident wrong number every time (the model converged hard on a bad pattern) or a different confident wrong number each time (there was nothing solid to converge on at all). Neither shape announces itself in the text. The prose reads the same either way — this is the uncorrelated-signals point from the wins list, and it's the reason you can't eyeball your way out of this.

3What actually reduces it, and what doesn't

Two moves that don't work, because they attack the wrong layer:

A bigger model. Scale helps with facts that are well-represented in training data — it doesn't touch the evaluation incentive that rewards guessing over abstaining on facts that aren't. A larger model hallucinates less on common knowledge and can still hallucinate confidently on anything sparse, for the identical reason a smaller one does.

Lower temperature. Sampling, and the knobs providers are taking away explained what temperature actually rescales — the shape of the distribution over tokens the model was already going to produce. It doesn't change which fact the model believes, because there is no separate belief store underneath the distribution to correct. A wrong answer at temperature 0 is just a more deterministic wrong answer.

What does help, none of it total:

• Self-consistency. Sample several times and check agreement, the way the code above does. This doesn't fix the underlying incentive, but it gives you an external signal — disagreement across samples — that the tone of any single answer can't provide.
• Ask for verification, not just an answer. A prompt that asks the model to state a claim and say whether it can point to a specific supporting source tends to surface hedging that a plain "answer the question" prompt suppresses, because you've changed what a good answer looks like, not just asked more nicely.
• Grounding, when you have something to ground against. If there's a document that should contain the answer, Citations and grounding is the actual fix — require a citation, verify the quoted span really appears in the cited chunk. That closes the gap for a different reason than anything in this lesson: it's not making the model more honest, it's giving you a mechanical check that doesn't rely on the model's honesty at all.
• Explicitly rewarding "I don't know." If you control any fine-tuning or evaluation of the model's outputs, score an honest abstention above a wrong guess. This is the only move that touches the actual mechanism from section 2 — it changes the incentive rather than working around it.

The practical shape of this lesson is a subtraction: stop trusting tone as evidence, since fluency was never a proxy for truth — only for "this continuation was likely," and a wrong answer can be exactly as likely-looking as a right one. When you have a document to check against, check against it. When you don't, sample more than once and look for disagreement. Neither is a full fix, because there isn't one — this is what the model does, not a state it can be talked out of.