Rate limits and reliability

Everything up to here has been about the model. This lesson is about the gate in front of it. Providers cap how fast you may call and how much you may spend, and they report both with the same status code. Every limit, tier name and field below is dated "as of Oct 2026", and each provider's own page is the authority, so the lesson tells you where to read the real number.

1.Three kinds of limit, and one that is not a limit

The names differ, but all three providers meter you on the same three shapes. As of Oct 2026:

Notice what is missing. None of the three pages documents a concurrency limit as of Oct 2026, apart from Google's cap of 100 concurrent batch requests. Concurrency is a tool you apply to yourself, a semaphore that caps in-flight calls, so a fan-out across a list does not spend a minute's budget in one second.

Three details change how you plan.

• A bucket, not a clock. As of Oct 2026, Anthropic says it uses a token bucket: capacity refills continuously instead of resetting on the minute. A limit of 60 per minute may be enforced as one per second, and limits are maximums, not guarantees.
• Ramps count. A sudden jump can earn a 429 under the stated numbers. As of Oct 2026, OpenAI's page says that past 1 million input tokens per minute you should grow by no more than 50% every 15 minutes, and that the threshold varies by model.
• Pools, and what counts. As of Oct 2026, Anthropic gives each model class its own limit, and a few older models share one bucket. For most models, cache reads do not count toward input tokens per minute, and max_tokens does not count toward output tokens per minute, only the tokens actually produced. This is not confirmed for OpenAI or Google in the sources used, so read their pages.

A worked example. The numbers are illustrative, so read your own from the provider's console. Say your tier allows 50 requests and 100,000 input tokens per minute. You must summarise 600 contracts. Each request carries a 20,000-token instruction block and a 10,000-token contract.

• Requests alone would finish in 12 minutes. Tokens do not: 30,000 per request gives 3.3 requests a minute, so 600 take about 180 minutes. Calling faster changes nothing, because tokens are the binding limit.
• Cache the instruction block on a provider that excludes cache reads from the limit, and only 10,000 count per request. That is 10 a minute, so about 60 minutes. The same work finishes three times sooner with no tier change. The first request pays to write the cache, and as of Oct 2026 Anthropic counts cache writes, so add a few minutes at the start.
• On a provider that counts cached tokens, that saving does not appear. The cache rules are in Caching and batch: the two big discounts.

Last, throttle early. As of Oct 2026, Anthropic returns anthropic-ratelimit-*-remaining and -reset headers (remaining counts are rounded to the nearest thousand), and OpenAI returns x-ratelimit-remaining-requests and x-ratelimit-remaining-tokens. Slow down as they fall, instead of sprinting into the wall.

2.One status code, two meanings

A 429 means "no, not now". Whether "now" ever ends depends on which limit you hit. As of Oct 2026:

As of Oct 2026, on Anthropic the spend-cap 429 has the same rate_limit_error type as a rate-limit one. What differs is error.details.error_code, which reads enforced_spend_limit_reached, and the missing retry-after header. Retries, including the SDK's automatic ones, fail until 00:00 UTC on the first of next month. As of Oct 2026, OpenAI also returns 429 at a hard spend limit, and its page says not to retry quota or billing errors. Google returns 429 RESOURCE_EXHAUSTED for a breach. As of Oct 2026, its page describes the spend limit as checked over a 10-minute window and says to wait and retry, and it gives no separate field to tell the cases apart, so log the whole body.

How it fails. A nightly job reaches the cap on the 24th. Your retry set contains 429, so each document gets five attempts with backoff, and the SDK's own retries (two by default on Anthropic's, as of Oct 2026) sit under each of those. The live feature, which shares the organisation's cap, is down too. The cure is a check before you retry. As of Oct 2026, you can also put a noisy batch job in its own Anthropic workspace with a lower limit, so it hits its own wall first.

For genuine rate limits, the backoff is exponential with full jitter: wait up to base * 2**attempt, picking a random time between zero and that. The jitter is the part people skip, because clients that failed together would otherwise retry together and rebuild the spike.

Four rules around it:

• Treat retry-after as a minimum. As of Oct 2026, OpenAI's page advises adding a small random delay on top, and falling back to jittered backoff when the header is missing.
• Cap the attempts and the total wait. An unbounded loop turns a thirty-second wobble into an hour of burned quota and hung connections.
• Keep a retry budget. As of Oct 2026, failed requests still count toward OpenAI's per-minute limits, so hammering does not help. If retries are a large share of your traffic, shed load.
• Know what your SDK already does. As of Oct 2026, Anthropic's SDK retries twice by default, honouring retry-after. A hand-rolled loop on top multiplies the attempts, so count them or switch the SDK's off.

Before you retry a 429, ask whether waiting can fix it. A rate limit says yes. A spend cap says no.

3.The loop, and when to leave it out

This loop tells the two 429s apart, honours retry-after, and jitters when the header is missing.

import random, time
from anthropic import Anthropic, APIStatusError, APITimeoutError

MODEL = "claude-sonnet-5-5"   # check the models page; as of Oct 2026

client = Anthropic(max_retries=0, timeout=60.0)   # this example owns its own retries

RETRYABLE = {408, 429, 500, 502, 503, 504, 529}

def is_spend_cap(err: APIStatusError) -> bool:
    body = err.body if isinstance(err.body, dict) else {}
    inner = body.get("error", body)               # some SDK versions unwrap "error"
    details = inner.get("details") or {}
    return details.get("error_code") == "enforced_spend_limit_reached"

def ask(prompt: str, model: str = MODEL, attempts: int = 5, max_wait: float = 120.0) -> str:
    deadline = time.monotonic() + max_wait             # total wait budget across all retries
    for attempt in range(attempts):
        try:
            resp = client.messages.create(
                model=model,
                max_tokens=2000,                       # thinking tokens share this budget
                messages=[{"role": "user", "content": prompt}],
            )
            # A thinking block may come first, so pick the first text block.
            return next(b.text for b in resp.content if b.type == "text")
        except (APIStatusError, APITimeoutError) as err:
            status = getattr(err, "status_code", 408)
            if status == 429 and is_spend_cap(err):
                raise                                  # waiting cannot fix a spend cap
            if status not in RETRYABLE or attempt == attempts - 1:
                raise                                  # a 400 dies here, correctly
            response = getattr(err, "response", None)
            header = response.headers.get("retry-after") if response else None
            try:                                       # the provider knows more than our formula
                pause = float(header) + random.uniform(0, 1)
            except (TypeError, ValueError):            # header missing, or an HTTP-date
                ceiling = min(60.0, 2.0 ** attempt)    # exponential
                pause = random.uniform(0, ceiling)     # full jitter: the part people skip
            if time.monotonic() + pause > deadline:
                raise                                  # this wait would blow the total budget
            time.sleep(pause)
    raise RuntimeError("unreachable")

I ran the retry logic with anthropic 1.15.0 against a stub server that returned the documented 429 bodies, not against the live API. The stub sent a thinking block before the text block, and the loop read the text. A spend-cap response raised after one call. Two rate-limit responses with retry-after: 0 were followed by success on the third. A retry-after: 500 with max_wait=10 raised at once. Check the shape of err.body on your own installed version, because is_spend_cap accepts both the wrapped and unwrapped forms. The max_retries=0 is there only so this loop does the work, and retrying a timeout is safe here only because this call has no side effect. In most applications let the SDK do its own backoff, and spend your effort on the next lesson, Timeouts, fallbacks, and failures that look like success.

When this does not apply.

• Low volume. A few calls a minute never meets a limit, and the SDK's defaults are enough.
• Batch jobs. The Batch APIs queue the work on the provider's side, so the limits become queue sizes. As of Oct 2026, Anthropic caps batch requests in the queue, and OpenAI caps queued input tokens per model. You poll for results and pace nothing.
• Self-hosted models. There is no provider cap. The limit is your GPU, and the 429 is one you write yourself. See Running a model yourself.
• Constant 429s. If limits bite every day, a better loop only hides it. Ask for a higher tier, or cache the repeated prefix.