Context windows and their limits

The window is the limit that returns an error before you get an answer. It is also the number people most often mistake for a quality setting, and it is not one. This lesson is about living inside it well. It assumes you have read What a model is made of, which explains tokens.

1.What has to fit, and what happens when it does not

The context window is a token budget for a single request. Not for a conversation, but for one call. As of Oct 2026, Anthropic's documentation counts everything in the request against it: the system prompt, every message, tool results, images, documents and tool definitions. The answer counts too, and so does any thinking.

As of Oct 2026, Anthropic lists a 1,000,000-token window and a 128,000-token maximum output for each of its four current models. Other providers publish their own figures, so read their model pages. Do not hard-code it. As of Oct 2026, the Models API (GET /v1/models) returns max_input_tokens and max_tokens for every model, so your code can ask. The token-counting endpoint gives an estimate of a request's size in tokens before you send it. As of Oct 2026 it is free, with its own rate limit. Count against the model you will actually use. As of Oct 2026, Anthropic says its newer tokenizer gives about 30 percent more tokens for the same text than older models, so counts from an older model mislead.

What happens at the edge has changed. As of Oct 2026, if the input alone is over the window, the API returns a 400 error, "prompt is too long". If the input fits but input plus max_tokens does not, current models (Claude 4.5 and newer) accept the request and stop mid-answer with stop_reason: "model_context_window_exceeded". Older models rejected that request instead (Anthropic's context windows page, as of Oct 2026). Either way, the API does not drop old messages for you.

So you need a policy before a user finds the edge: drop old turns, summarise them, or retrieve only the parts that matter. Anthropic now offers server-side compaction, which writes the summary for you. It was in beta as of Oct 2026.

2.A bigger window is not a free upgrade

A larger window costs you several things.

You pay for every token, every turn. A 100,000-token prompt costs the same whether the answer needed all of it or 400 tokens of it, and again on the next turn. Anthropic bills the full 1M window at standard rates on most current models, with one exception where prompts over 100,000 tokens cost more (its context windows page, as of Oct 2026).

Latency goes up before generation starts. Time to first token is mostly the model reading your input, and attention work grows with the square of the length (see What a model is made of). A very long prompt means a blank screen.

Accuracy can fall as the prompt grows. Anthropic's own documentation calls this context rot: recall and accuracy degrade as token count rises (as of Oct 2026). Three pieces of evidence show how it looks.

• Position. Liu and colleagues (2023) found models retrieve facts near the start or end of a long prompt more reliably than the same fact halfway through. They called it lost in the middle.
• Length. Chroma tested 18 models in July 2025, none of them current. Accuracy fell as input grew, even on simple tasks. In their plain planted-fact test, position made no notable difference.
• Look-alike text. In the same report, one on-topic passage that did not answer the question lowered accuracy, and four lowered it more.

Chroma sells a vector database, which suits the conclusion "retrieve less", and its models are old, so treat this as a reason to test, not a measurement of your model. One more result: models given about 300 tokens of focused context beat the same models given a roughly 113,000-token history, on the same questions.

Picture this, as an illustration and not a logged incident. An HR assistant gets three handbook versions pasted into one prompt "to be safe": 2023, 2024 and current. Someone asks how much notice they must give. The right clause is in the current version, halfway down. Two older clauses read almost the same and give different numbers. The model answers confidently with the 2023 number. Nothing errored. The answer is wrong, and it looks right.

A bigger window raises the ceiling. It does not raise the accuracy. What you put in the window still decides the answer.

3.Working well inside it

Five moves, in the order they pay.

Put the long text first and the question last. Anthropic's guidance for long prompts, as of Oct 2026: put long documents near the top, above your query, and put the query at the end. It reports gains of up to 30 percent in its own tests, especially with several documents. That is a vendor figure, so test it on your task. Wrap each document in tags with its source, so the answer can cite it.

<documents>
  <document source="handbook-2025.pdf">...40,000 tokens...</document>1
</documents>

Answer only from the documents above. Cite the source you used.2

How much notice do I have to give?3

Retrieve rather than paste. Ten relevant paragraphs beat the whole handbook on cost and latency, and by the evidence above often on accuracy as well. The RAG course covers building it. Keep the chunk count small, because every extra chunk is another chance for a near-miss.

Count the answer. Output shares the window, and as of Oct 2026 Anthropic's context windows page says thinking tokens are a subset of max_tokens. A small max_tokens on a model that thinks can be used up before any visible text appears. Size it for the thinking and the answer together.

Cache what never changes. A long, fixed prefix can be reused at a fraction of the price. The rules for that are in Caching and batch: the two big discounts.

Then measure. Take your real prompt length and plant the true answer at several positions. Beside it, plant a near-miss: a sentence that looks like the answer and is wrong. A fact alone is the easy case. Run it on the model you actually use.

import random
import re
from anthropic import Anthropic

MODEL = "claude-sonnet-5-5"   # check the models page; valid as of Oct 2026
client = Anthropic()

PARAS = open("handbook.md").read().split("\n\n")   # your own text, at your real length
NEEDLE = "Notice to resign is 30 days."
NEAR_MISS = "Notice to end a lease early is 90 days."
ASK = "How many days' notice do I give to resign? Reply with the number only."

def hits(at, runs=5):
    ok = n = 0
    for _ in range(runs):
        p = PARAS[:]
        p.insert(random.randrange(len(p) + 1), NEAR_MISS)
        p.insert(int(at * len(p)), NEEDLE)   # needle placed last, so it sits at `at`
        r = client.messages.create(
            model=MODEL, max_tokens=4000,   # thinking shares this budget
            messages=[{"role": "user", "content": "\n\n".join(p) + "\n\n" + ASK}],
        )
        if r.stop_reason == "max_tokens":
            continue   # starved of room, not a miss
        n += 1
        text = "".join(b.text for b in r.content if b.type == "text")
        ok += bool(re.search(r"\b30\b", text))
    return f"{ok}/{n}"

for at in (0.0, 0.25, 0.5, 0.75, 1.0):
    print(f"{at:.0%}: {hits(at)}")

You get five lines, one per position, each showing hits out of the runs that finished. A dip in the middle is the finding, and so is a dip everywhere; a run cut off by max_tokens is skipped, not counted as a miss. Run it again without the NEAR_MISS line to see what one look-alike sentence costs. Mind the bill: it sends 25 requests at your full prompt length, so count the tokens, multiply by 25 and check the provider's pricing page.

When not to use any of this. If the whole document is the task, such as summarising one 300-page report or finding contradictions across a codebase, retrieval would cut the links you need. Use the big window and test it. If your prompts run to a few thousand tokens, none of this matters yet, so do not optimise. And if your own test finds the planted fact everywhere, trust that result over every rule here, with its date and its model name attached.