What a model is made of

You can rent one of these for a fraction of a cent and never think about what it is. That works until you must choose between two of them, or work out why a few thousand extra words cost more than you budgeted. This lesson opens the box far enough to answer those.

1.It is a file of frozen numbers

A model is a large array of numbers called weights. Training sets those numbers. Serving only reads them. Once a model ships, the file does not change, not while it answers you and not after a long conversation.

Three things follow:

• It cannot learn from your conversation. What you send is input for one request and then it is gone. Whether your data is stored, or later used for training, is a policy question about the provider. It is a different question from this one.
• There is no lookup table inside. Facts are not rows it reads. They are patterns baked into the numbers, which is why it can be confidently wrong about one.
• An updated model is a different file. Anthropic says the weights under a model id never change, and that an update ships under a new id (as of Oct 2026). So pin the full id in your code and re-run your tests when you move.

Pinning has a limit. The same page says the serving code around the weights (the router, the safety classifiers, the sampling logic) can change under an unchanged id, so behaviour can shift slightly without a new file. Keep a small test running after you ship. Timeouts, fallbacks, and failures that look like success covers how that shows up.

Parameter count is how many numbers are in the file, quoted in billions. It is the figure everyone reaches for, so it helps to know exactly what it buys.

When this does not apply. Two cases break the table. First, closed providers mostly do not publish parameter counts (as of Oct 2026, from what I could see; I did not survey every provider), so for those models there is nothing to compare. Second, in a mixture-of-experts model only a slice of the network works on each input. The total count then sets the memory you need, while the active slice sets the speed. What fits on a GPU works through what that means for hardware. In both cases, quality is better judged by testing than by counting.

2.Tokens in, attention inside, and why length costs more

Before any arithmetic, your text has to become numbers. A tokenizer cuts the string into chunks called tokens and looks each chunk up in a fixed vocabulary to get an integer. The model receives only those integers. The answer comes back the same way in reverse.

A token is usually a few characters. Sometimes it is a whole common word, more often a fragment of a longer one. The cuts are not where a reader would put them:

How many tokens is my text? There is no safe rule of thumb. Every model family ships its own tokenizer, and providers change them. As of Oct 2026, Anthropic says its newer tokenizer gives about 30% more tokens for the same text than its older one, so a million tokens now holds roughly 555,000 words where it once held about 750,000 (those two figures imply nearer 35%, so count with the provider's tool). The fix is to ask the provider. Anthropic offers a token-counting endpoint, and as of Oct 2026 it is free to call (it has its own rate limit). The count is an estimate, and the real figure can differ by a small amount. The code at the end of this lesson uses it.

To watch the cuts happen, try Tiktokenizer. It is a third-party page, and I could confirm only that it lists an OpenAI tokenizer, so its counts will not match Claude's. Use it to see the idea, not to budget.

Failure case: the model cannot see letters. Ask a model how many times r appears in "raspberry". The word arrives as a few tokens, not as a row of characters, so the model has no clean way to count inside it. It can answer "2" with complete confidence. The same blindness hits reversing a string, checking a spelling and adding two long numbers exactly. Push that work into code, such as a tool call or a few lines of Python, instead of asking the model to look at something it cannot see.

Tokens are also the unit you are billed in and rate-limited in, and the wait before the first word is mostly the model reading your input tokens. So you budget in tokens, not words.

The architecture that consumes those tokens is a transformer: a stack of identical blocks, each doing an attention step and then a small network applied to each token on its own. In attention, every token scores itself against every other token and takes in a weighted blend of what is relevant. Stack that step dozens of times and each token has absorbed its context. That is how the model knows that it means the database and not the query.

Now count the comparisons. Every token against every other token means n tokens give about n² pairs.

Attention compares every token with every other token. Double the prompt, and you roughly quadruple that part of the work.

Do not expect a clean curve when you measure. The per-token network grows linearly and dominates at short lengths, and serving stacks fight the quadratic with caching and fast attention kernels. So for a short prompt this rule hardly matters. For a long one the direction never reverses: a long prompt is slow to read before anything else.

3.Where behaviour comes from: two training stages

Almost everything you like or dislike about a model was decided in one of two stages, and usually the second.

Pre-training. Read a very large pile of text, predict the next token, adjust the weights when wrong, and repeat on a large cluster of accelerators. This takes most of the money. What comes out is a base model. It has absorbed grammar, facts, code and style, and it can do exactly one thing: continue text.

Post-training. Much smaller and cheaper, and it decides nearly everything you interact with. First comes supervised fine-tuning on written demonstrations of good answers. Then comes preference tuning, where people (or another model working from written rules) rank candidate answers and the weights move toward the preferred one. Instruction following, tone, refusals, formatting habits and tool use come from here.

So why do two models of the same size behave so differently? Not architecture, which is public and similar across the frontier labs. The differences are:

• Data. What was collected, how it was filtered, how much code and maths is in the mix. Labs guard this closely.
• The training budget. How much data each parameter saw. The Chinchilla paper argued that model size and training tokens should grow together, so size alone was never the whole story. It is one input into why small models got good, not the full account.
• Post-training. The demonstrations, the preference data and the rules. Two labs can start from near-identical base models and ship products that feel nothing alike.

Which leaves one instruction: you cannot read quality off a spec sheet. Not from parameter count, and not from a leaderboard built on someone else's tasks. Write twenty examples from your own workload, with the answers you want, and run the candidates against them. That is a morning of work and it outranks every number a provider publishes.

Here is the worked example. It counts tokens for a repeated sentence at three sizes, using the provider's counter and not a rule of thumb.

# pip install anthropic   (checked against SDK 1.15.0)
from anthropic import Anthropic

MODEL = "claude-sonnet-5-5"  # check the models page; as of Oct 2026

client = Anthropic()
sentence = "The quick brown fox jumps over the lazy dog. "

for repeats in (1, 4, 16):
    text = sentence * (100 * repeats)
    count = client.messages.count_tokens(
        model=MODEL,
        messages=[{"role": "user", "content": text}],
    )
    print(repeats, "x:", count.input_tokens, "tokens for", len(text), "characters")

I wrote this against the documented call and the SDK's own signature, but I did not run it against the live API, so the numbers are yours to read. Expect the count to grow roughly in line with the repeats. Then change MODEL to an older id that is still listed as active and compare the two counts for the same text. That gap is the tokenizer, and it is why you recount before a move. The script counts input only; if you time a real request, thinking tokens add to the wait, as Thinking and effort, model by model explains.