What the model actually is

You know how a database works, roughly, even if you have never written one. That mental model is what stops you writing a query that scans a billion rows by accident. This lesson gives you the same thing for a language model. It is a smaller machine than the marketing suggests, and once you can see it, most of its strange behaviour becomes predictable.

1It reads tokens, and tokens are not words

A token is a chunk of text — usually a few characters, often a word fragment. Before the model sees your prompt, a tokenizer chops it into these chunks and turns each one into a number. The model only ever deals in those numbers. It never sees your string.

The split is not where you would put it:

For ordinary English, the rule of thumb is about 4 characters per token, or roughly 3 tokens for every 4 words. For Japanese, Hindi or Arabic the same sentence can cost two to four times more tokens. If you are billing per user, users writing in those languages cost you more for identical work.

Two practical consequences fall straight out of this.

The model is blind to letters. It cannot reliably count the r's in strawberry, reverse a string, or tell you the third character of a word, because letters are not the unit it reads. Push that work into code, where it belongs.

Tokens are your unit of budget. Price is per token, split into input and output rates, and output is usually the more expensive of the two. Speed follows the same unit: input is processed in roughly one pass, but each output token is produced one at a time, so a 2,000-token answer takes about ten times longer to come back than a 200-token one. When you want a faster endpoint, shorten the answer before you shorten anything else.

2The context window is the only memory there is

The API is stateless. It stores nothing about you between calls. The model has no memory of your last request, and no memory of the last ten thousand.

What looks like a conversation is a trick your code performs. Every turn, you send the entire history — system prompt, all previous user messages, all previous assistant replies, plus the new message — and the model reads the whole pile fresh, answers, and forgets it all again.

The context window is the hard ceiling on that pile, measured in tokens. Input and output both have to fit inside it. When a long chat starts failing, this is almost always why.

The model has no memory. Your code has the memory, and it re-reads the whole thing out loud on every single turn.

Three things follow that will save you real money:

• Long chats get expensive in a curve, not a line. Turn 30 resends turns 1 through 29. Cost per turn climbs the whole way.
• You need a plan for the ceiling. Drop old turns, summarise them into one message, or retrieve only the relevant parts. Pick one deliberately before a user hits it for you.
• Repeated prefixes can be cached. If the first 10,000 tokens of every request are the same system prompt and documents, prompt caching lets the provider skip re-reading them and charges much less for that part. It is the single easiest cost win in most apps.

3Every answer is a guess, and there is a dial on it

Given everything so far, the model produces a probability distribution over the next token. Then something picks one. Then the whole thing runs again with that token appended. That loop is the entire show.

The picking step is sampling, and you control it:

This is why the same prompt gives different answers. Nothing is broken. At any temperature above 0 you are drawing a sample, and different draws give different text. Even at temperature 0, floating-point and infrastructure details mean you can get small differences between runs, so treat 0 as much more stable rather than guaranteed identical.

And this is what a hallucination really is. There is no lookup step. The model is not consulting a store of facts and occasionally reading the wrong row — there is no store and no reading. It is predicting text that fits the pattern, and text that fits the pattern is often true, because true things are common in what it learned. When it is not true, the machine did exactly what it always does. That is why "be accurate" in a prompt does so little, and why giving it the actual document (lesson 4) does so much.

# pip install anthropic
from anthropic import Anthropic

client = Anthropic()

prompt = "List three causes of a slow SQL query. One line each."

# Ask before you pay: count the input tokens for the exact request you are about to send.
count = client.messages.count_tokens(
    model="claude-sonnet-5",
    messages=[{"role": "user", "content": prompt}],
)
print("input tokens:", count.input_tokens)

resp = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=200,      # seatbelt: caps cost and latency
    temperature=0,       # repeatable-ish: take the most likely token every step
    messages=[{"role": "user", "content": prompt}],
)

print(resp.content[0].text)
print(resp.usage)        # what you were actually billed for, input and output

Run it twice at temperature=0 and then twice at temperature=1, and watch the difference. That two-minute experiment is worth more than any explanation of sampling, including this one.