Why models hallucinate

A model that does not know a fact still hands you an answer, in the same calm tone it uses for a fact it does know. This lesson explains why, using one 2025 paper from OpenAI researchers. It then covers what to do about it, and where that advice stops working.

1.Answering is the default, and doubt gets no turn

Every answer comes out of one step. A forward pass ends in a distribution over the next token, and one token is sampled. Sampling, and the knobs providers are taking away covered that step. What matters here is what it lacks: nothing in it makes the model check how sure it is before it answers.

A refusal is not a separate mode that sits outside generation. "I'm not certain about that" is a run of tokens, and it must win the same contest as every other continuation. A specific, fluent answer often scores higher.

Be careful with the next step, because it is easy to overstate. The model may well carry some sense of doubt. Kadavath and colleagues (2022) found that large models were mostly well calibrated on multiple-choice and true/false questions. Kalai and colleagues report that pre-trained models are often calibrated too, and that reinforcement-learning post-training can make it worse. So the claim is not "no doubt signal exists". The claim is that nothing forces the answer to follow it.

A wrong answer and a right one come off the same production line. Nothing in the wording tells you which is which.

2.Guessing is what the scoring pays for

Kalai, Nachum, Vempala and Zhang (OpenAI, September 2025) give two causes. Both matter.

The first is a floor on rare facts. Facts that appear often and consistently are learned well. A fact seen exactly once, such as one person's birthday, gives the model nothing to prefer the right answer over a plausible wrong one. The paper shows the error rate on such facts is at least about the share of facts seen once. If a fifth of birthdays appear once in the data, expect at least about a fifth of them wrong. It also shows that the error rate of generating an answer is at least about twice that of judging whether a given answer is valid. This is theory, not a measurement of any trained model.

The second is the grading. Most benchmarks work like an exam with no penalty for a wrong guess. A right answer earns a point, and a wrong answer and "I don't know" both earn zero. Under that rule guessing never loses. The authors say models are "optimized to be good test-takers", and they audited popular benchmarks and judge models to see whether this holds. As of mid-2025, nine of the ten it audited gave no credit for "I don't know", including GPQA, MMLU-Pro, IFEval and SWE-bench. One gave partial credit. I did not re-check today's versions, so look at the scoring rule of any benchmark you rely on. Reading a leaderboard returns to this.

The authors' fix is to state a confidence target t in the prompt or the grader, and to charge t/(1-t) points for a wrong answer. At t = 0.5 a wrong answer costs 1 point, and at t = 0.9 it costs 9. Answering then pays only when the model's chance of being right beats t.

3.What helps, what does not, and where it stops

Two common fixes attack the wrong layer. A bigger model helps with errors of capacity and with well-covered facts. It does not remove the floor on rare facts, and it does not change what the grading pays. A lower temperature only makes the answer steadier: a wrong answer stays wrong, and as of Oct 2026 newer Anthropic models reject a non-default value anyway (the lesson cited in section 1 has the detail). A reasoning model (see Base, instruct, and reasoning models) fixes errors of working, like miscounting letters, because the written trace spells the word out. It cannot supply a fact it never learned.

What does help is partial, and each item has a limit:

• Allow and score "I don't know". As of Oct 2026, Anthropic's guide tells you to give the model explicit permission to admit uncertainty. In your own evals, score an abstention above a wrong answer.
• Ground it. If a document should hold the answer, require a quote and check that the quote appears in it. Citations and grounding, in the RAG course, covers this.
• Sample several times and compare. Disagreement is a signal that a single answer's tone can never give you.

Here is a script that tries two of these. It asks one obscure question five ways with a plain prompt, then five with the paper's idea written into the prompt. Swap in a question whose answer you can verify.

from collections import Counter

import anthropic

MODEL = "claude-sonnet-5-5"  # check the models page; as of Oct 2026

client = anthropic.Anthropic()

QUESTION = "What was the exact attendance at the 1987 IEEE conference on parallel computing?"

PLAIN = QUESTION + " Reply with the number only."

# t = 0.9, so a wrong answer costs t / (1 - t) = 9 points.
SCORED = (
    QUESTION + " Reply with the number only, or the word unknown.\n"
    "Scoring: correct = +1, unknown = 0, wrong = -9. "
    "Answer only if you are more than 90 percent sure."
)

def ask(prompt):
    resp = client.messages.create(
        model=MODEL,
        max_tokens=4096,  # covers thinking plus the answer on this model
        messages=[{"role": "user", "content": prompt}],
    )
    # A reply can start with a thinking block, so pick the text blocks by type.
    return "".join(b.text for b in resp.content if b.type == "text").strip().lower()

for name, prompt in [("plain", PLAIN), ("scored", SCORED)]:
    print(name, Counter(ask(prompt) for _ in range(5)))

Run it and read the two counters. You may see the same number five times, a different number each time, or "unknown". Each of those is a different diagnosis, and none announces itself in the wording. On this model, as of Oct 2026, a request with no thinking field runs with thinking on, which is why the script reads blocks by type and gives max_tokens room; the Sonnet 5.5 migration guide covers this. I could not confirm what this model prints for this question, so treat the result as yours to read.

A failure to expect. Picture a support bot with retrieval. A customer asks about returning an opened blender after 45 days, and the search finds nothing relevant. The prompt says "answer the customer", so the model writes a clear, friendly policy that nobody wrote. This case is made up, but it is the typical shape. Retrieval did not fail loudly, and the model had no instruction for an empty result. The fix is to tell it what to say then, and to put an empty-retrieval case in your tests.

Where this advice does not apply.

• You hold the source. Do not ask the model to recall a fact you can look up. Check it against the source.
• The mistake is in the working. Counting, arithmetic and long chains need a reasoning trace or a tool. A permission to abstain will not fix them.
• Agreement is not truth. A model can settle on the same wrong answer every time. Five samples also cost five calls.
• The scored prompt is a prediction. A model told to be 90 percent sure may refuse questions it could answer, so test it on cases with known answers and count the refusals. In Chroma's July 2025 test of 18 models, none current, one family abstained more and another gave confident wrong answers more. That is a snapshot, not a trait of a vendor, and Chroma sells a vector database.