Thinking out loud

This is the most over-applied idea in prompting. "Think step by step" gets pasted into prompts that have nothing to think about, and the result is slower, more expensive, and no more correct. The underlying effect is real. The mechanism behind it tells you exactly where it applies.

1The mechanism, not the magic

The model does a fixed amount of computation per token. One forward pass, one token out. That single fact explains chain-of-thought completely.

A task needing four separate derivations cannot be done in the one forward pass that produces the answer token. There is not enough computation there. Make the model write the four steps first and it gets four passes' worth of room — and, just as importantly, each written step lands in the context, so the next step is conditioned on a partial result rather than on nothing. It is a scratchpad, not an inner voice.

Two consequences fall out immediately, and both are practical.

The reasoning has to come before the answer. Ask for the answer and then the explanation, and the answer token is generated with no working available, exactly as if you had never asked. Worse, everything after it is written to fit a conclusion already in the context. You get a justification, not reasoning. This ordering bug is common and it silently removes the whole benefit.

More tokens is the point. "Take your time" and "be careful" produce no additional tokens, so they produce no additional computation — which is why lesson 1 puts them in the do-nothing pile. "List the constraints one per line, then answer" produces tokens. That is the difference between the two phrasings, and it is the only difference that matters.

One honest caveat before you build on this. The written reasoning is not a reliable account of how the answer was produced. Turpin et al. (2023) showed models giving fluent step-by-step explanations that did not mention the feature actually driving the answer. So the working is genuinely extra computation, and genuinely useful for debugging, but it is not evidence.

The written reasoning is extra computation, not a confession. Do not use it as an audit trail.

2Where it pays, and where it is waste

A quick test that gets it right most of the time: if you could do the task yourself without writing anything down, the model probably can too.

Now the cost, plainly. Reasoning tokens are output tokens. Output tokens are the expensive ones, and they are produced one at a time, so they dominate latency (build-with-llms lesson 1). A 300-token derivation in front of a five-token answer means roughly sixty times the output of the answer alone, in both money and wall-clock time. In a nightly batch job that is irrelevant. In something a user is watching, it is the whole experience.

And the honest note this area badly needs: "chain-of-thought helps" is a statement about populations of tasks, not about yours. The published gains are on reasoning benchmarks. On your classification, with your prompt and your data, it may do nothing at all, or it may help on one input class and hurt on another. Run it against a fixed test set both ways before you ship it — build-with-llms lesson 8 covers how to build that set. This is one of the few prompting decisions that is cheap to measure and expensive to guess at.

3The practical shape: a scratchpad you strip

Put the reasoning in a tagged block and remove it before anything reaches the user. Two reasons: it never leaks into the product, and a tag is trivial to split on in code.

import re
from anthropic import Anthropic

client = Anthropic()

SYSTEM = """Decide whether an expense is reimbursable.
Work inside <scratch> tags first: name the policy rule, then the fact that
triggers it, then the conclusion.
After </scratch>, output exactly one word: approve, reject, or escalate."""

def decide(expense: str) -> tuple[str, str]:
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=400,          # room for the working, not just the word
        temperature=0,
        system=SYSTEM,
        messages=[{"role": "user", "content": expense}],
    )
    text = resp.content[0].text
    found = re.search(r"<scratch>(.*?)</scratch>", text, re.S)
    answer = re.sub(r"<scratch>.*?</scratch>", "", text, flags=re.S).strip()
    return answer, (found.group(1).strip() if found else "")

answer, working = decide("Dinner for 4 with a client, 180 EUR, Tuesday.")
print(answer)     # goes to the user
print(working)    # goes to your logs

Note what the system prompt asks for: rule, then fact, then conclusion. Not "think about it". Naming the steps is the difference between reasoning that helps and three sentences of throat-clearing that cost the same.

Keep the working in your logs. It is the best debugging artefact this whole course offers — even though it is not a faithful trace, it shows which rule the model latched onto, and that is usually enough to find the ambiguity in your policy.

Reasoning models. Some models do this internally: you set a thinking budget, they reason, and the API returns the thinking separately from the answer. Where that is available, use it instead of hand-rolling tags. You get the separation for free, the model was trained to use that budget well, and you are not spending your own prompt on scaffolding.

On those models, adding "think step by step" is usually redundant, and sometimes actively worse — you get a second, shallower pass layered on top of the real one, and a longer answer for no gain. What still helps is telling it what to consider: the constraints, the edge cases, the rule that trips people up, the thing you always find yourself checking by hand. That is information the model does not have. "Think" is information it already had.

The reasoning is sorted. The next lesson deals with what comes out the other end, and how to stop it being a surprise.