Thinking out loud

This is the most over-applied idea in prompting. "Think step by step" gets pasted into prompts that have nothing to think about, and the result is slower, more expensive, and often no more correct. The effect is real. The mechanism tells you where it applies, and why current models have mostly taken the job off your hands.

1.The mechanism, not the magic

The model does a fixed amount of computation per token. One forward pass, one token out. That single fact explains chain-of-thought completely.

A task needing four separate derivations cannot be done in the one forward pass that produces the answer token. Make the model write the four steps first and it gets four passes' worth of room, and each written step lands in the context, so the next step is conditioned on a partial result. It is a scratchpad, not an inner voice.

Two practical consequences fall out.

The reasoning has to come before the answer. Ask for the answer and then the explanation, and the answer token is generated with no working available. Everything after it is written to fit a conclusion already in the context. You get a justification, not reasoning, and the whole benefit silently disappears.

More tokens is the point. "Take your time" and "be careful" produce no additional tokens, so they produce no additional computation — which is why Why prompts fail puts them in the do-nothing pile. "List the constraints one per line, then answer" produces tokens. That is the whole difference.

One honest caveat. The written reasoning is not a reliable account of how the answer was produced. Turpin et al. (2023) showed fluent step-by-step explanations that never mentioned the feature actually driving the answer. The working is real computation and useful for debugging, but it is not evidence.

The written reasoning is extra computation, not a confession. Do not use it as an audit trail.

2.Where it pays, and where it is waste

A quick test that gets it right most of the time: if you could do the task yourself without writing anything down, the model probably can too.

Now the cost. Reasoning tokens are output tokens, the expensive kind, produced one at a time, so they dominate latency — the LLMs course explains why in What you actually pay for. A 300-token derivation in front of a five-token answer is roughly sixty times the output, in money and in wall-clock time. As of October 2026 you are billed for the full thinking even when the API shows only a summary. In a nightly batch that is irrelevant. In something a user is watching, it is the whole experience.

The evidence is thinner than the folklore. A 2025 Wharton study ran GPQA Diamond, 198 hard science questions, 25 times each (as of June 2025: older models, no Claude with thinking, one benchmark). Against a baseline telling the model to answer with no explanation, "think step by step" lifted the average for four of five non-reasoning models by 7 to 14 points. Against the model's own default the lift was small or absent, because those models already reason a little unprompted. So the baseline you pick decides the headline. The average also hid a cost: three of the five models got worse: the share of questions they answered correctly on every one of 25 trials fell by 9 to 17 points.

That is why "chain-of-thought helps" is a claim about populations of tasks, not about yours. On your data it may do nothing, or help one input class and hurt another. Test it both ways on a fixed set before you ship — Versioning and testing prompts, later in this course, builds that set.

3.Let the model think, then read the blocks

Hand-rolled reasoning is mostly retired. As of October 2026, thinking is on by default on Claude Sonnet 5.5, and the API returns it as separate blocks beside the answer. That is the section 1 rule with the plumbing done for you: the model works first, the answer comes after, and nothing leaks into the text a user sees. Your job shrinks to three things. Leave room for the working in max_tokens. Read the reply by block type, because the first block may be thinking, may be empty, or may not exist. And to log the working, ask for a summary, because the default omits it.

import logging

from anthropic import Anthropic

client = Anthropic()
log = logging.getLogger(__name__)
MODEL = "claude-sonnet-5-5"   # as of October 2026

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

def thinking_of(resp) -> str:
    return "\n".join(b.thinking for b in resp.content if b.type == "thinking")

SYSTEM = """Decide whether an expense is reimbursable under the policy below.
Check the per-head cap and the receipt rule before you decide.
Answer with one word: approve, reject, or escalate.

<policy>...</policy>"""

def decide(expense: str) -> str:
    resp = client.messages.create(
        model=MODEL,
        max_tokens=2000,   # thinking counts against this
        thinking={"type": "adaptive", "display": "summarized"},   # a summary to log
        system=SYSTEM,
        messages=[{"role": "user", "content": expense}],
    )
    if resp.stop_reason != "end_turn":
        log.error("raw reply: %r", text_of(resp))   # keep what came back
        raise RuntimeError(f"no usable answer: {resp.stop_reason}")
    log.info("thinking summary: %s", thinking_of(resp))   # for debugging, not proof
    return text_of(resp).strip()

print(decide("Dinner for 4 with a client, 180 EUR, Tuesday, no receipt."))

The parameter spellings are copied from the vendor docs as of October 2026, not run here, so try them on your SDK version first. The system prompt names what to check and never says "think". A one-word answer is safe here only because the built-in thinking supplies the room. Suppose the policy caps client meals at 40 EUR a head and wants a receipt above 25. The dinner for four has to be divided, compared to the cap, and checked for a receipt: three steps, and the decision needs all of them. That is the kind of task section 2 says pays for its tokens. The logged summary shows which rule the model latched onto, usually enough to find the ambiguity in your policy.

Failure case: the scratchpad that gets refused. The older shape, asking for working in <scratch> tags or a reasoning field of the reply, can be declined as reasoning_extraction as of September 2026, so do not ask for the reasoning in the output; Stop reasons and refusals has the trigger list and the billing.

Do not stack the old phrase on top. Vendor guidance prefers a general instruction over a hand-written plan, because the model's own plan often beats yours. The measured gain is small too. In the Wharton study the three reasoning models moved about three points with the step-by-step line, up for two and down for one, at 20 to 80 percent more time (as of June 2025). What the model lacks is your context: which constraints matter, which edge case bites, what you always check by hand. Say that. "Think" is information it already had. (JSON-only answers are the one exception; see Steering how much it thinks.)

One setting changed. As of September 2026, a fixed budget_tokens returns a 400 on Claude 4.7 and later, and on Sonnet 5.5 so does disabled. Effort replaces them; it is the next lesson.

When not to use it. Skip it where nothing is derived: extraction, formatting, translation, a lookup with one right word. Skip it where a user is waiting and the task is easy. Skip hand-written step plans on models that think natively. Never present the working as the reason for a decision. And do not swing the other way: a one-word format on a task that needs working out removes the room the model needed.

Next: how much thinking you get and what it costs, and why telling a model to think less often does not work — Steering how much it thinks.