Retries and fallbacks

Everything in Validating what comes back ends in an exception, and Stop reasons and refusals sorted the replies that have no body to check. That keeps the bad value out of your database, but your user still got nothing. This lesson is about the space between "we rejected it" and "we failed".

1.Sort the failure, then decide

Cap it at three calls in all: the first try plus up to two more, and a cut-off retry uses one of them. That number is this course's rule of thumb, not a measured one; the vendor's Sonnet 5.5 guide, as of October 2026, suggests a single retry for a failed field check. A third failure on the same input is not bad luck: the prompt asks for something the model does not reliably give. Let it fail and go and look.

Rate limits and overload need a delay, and the delay needs jitter: exponential backoff with a random part, so a thousand clients that failed together do not retry together. The LLMs course covers the headers in Rate limits and reliability; here, a 429 just gets its own branch, with a sleep in it.

A blind retry asks the same question again. A retry with the error attached asks a different, easier one.

2.Retry with the error attached

This is the biggest win in the lesson, and it is one line of code. A rejected reply comes with an error that says exactly what was wrong: unknown owner 'dave'. Append the failed reply and that error to the conversation and ask again. The model can usually fix a problem it can see.

A blind retry repeats the question and hopes for different sampling. The repair turn asks a narrower one: fix this specific thing, with the answer mostly written already. Two details make it work:

• Pass your own error text. Do not summarise it to "that was wrong". Pydantic and Zod messages name the field and the constraint.
• Do not re-send the instructions. They are already in the conversation.

Be honest about the size of this. Under a decoder-enforced schema the shape failures that made retry famous mostly vanish. What is left is meaning: an owner who does not exist, a quote not in the source. Those suit a repair turn, because the error is specific.

Append, never edit. A repair call that adds two turns at the end leaves the earlier request identical, so it can plausibly read the prompt cache from the first call. That is an inference from how prefix caching works, not something the vendor says about retries. Rewriting an earlier turn, or changing effort or thinking between calls, breaks the prefix, and editing history invalidates replayed thinking blocks: on Sonnet 5.5, for accounts created on or after 31 August 2026, the request returns an error.

If a loop is stuck anyway, change the request: attach the error, reword the instruction, reorder an example, or go up a rung of the ladder below.

3.The ladder, the bill, and one loop

When the retries run out, you need somewhere to go. Decide the order in advance, per feature:

1. Repair retry. Same model, error attached. Cheapest, fixes most things.
2. A different or stronger model. Worth a rung at the edge of the first model's skill, or when a refusal category allows it. Slower and dearer, and it starts without the first model's thinking: as of October 2026, blocks it cannot read are dropped silently.
3. A deterministic default. A regex, a lookup, or the safe value, such as owner: null.
4. Fail loudly. Raise, alert, put the case where a human will see it.

Never make rung four a silent success: an empty result with a 200 hides an outage until a report is wrong next month. Two guard rails go around it all. A budget cap: count attempts (and tokens, if you track them) per request and stop at a ceiling, because a retry loop over a large batch is how a small bug becomes a large invoice. Idempotency: if the answer triggers an effect, such as a message sent or a row written, a retry must not do it twice, so give each logical request a key and check it first. The model call is safe to repeat; what you do with its answer often is not.

import logging
import random
import time

from anthropic import Anthropic, APIConnectionError, APIStatusError
from pydantic import BaseModel, Field

client = Anthropic()
log = logging.getLogger(__name__)
MODEL = "claude-sonnet-5-5"   # as of October 2026
OWNERS = {"alice", "bob", "priya"}
MAX_ATTEMPTS = 3   # three calls in all; a cut-off retry uses one

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

class Ticket(BaseModel):   # the triage model from Validating what comes back
    urgency: int = Field(ge=1, le=5)   # checked here, because the decoder cannot hold a range
    owner: str | None
    quote: str

# By hand, not parse(), so the raw reply stays in your hands for the repair turn.
TICKET_SCHEMA = {
    "type": "object",
    "properties": {
        "urgency": {"type": "integer"},
        "owner": {"type": ["string", "null"]},
        "quote": {"type": "string"},
    },
    "required": ["urgency", "owner", "quote"],
    "additionalProperties": False,
}

def check(ticket: Ticket, source: str) -> None:
    # simplified; see squash() in Validating what comes back
    if ticket.owner is not None and ticket.owner.casefold() not in OWNERS:
        raise ValueError(f"unknown owner {ticket.owner!r}; use null if no listed owner is named")
    if ticket.quote not in source:
        raise ValueError("quote does not appear in the source text")

def triage(source: str) -> Ticket:
    messages = [{"role": "user", "content": source}]
    max_tokens = 2000   # thinking counts against this
    for attempt in range(MAX_ATTEMPTS):
        try:
            resp = client.messages.create(
                model=MODEL, max_tokens=max_tokens, system="Triage the ticket. Use null for an owner the text does not name.",
                messages=messages,
                output_config={"format": {"type": "json_schema", "schema": TICKET_SCHEMA}},
            )
        except (APIConnectionError, APIStatusError) as err:   # connection errors include timeouts
            transient = isinstance(err, APIConnectionError) or err.status_code in (429, 500, 503)
            if transient and attempt < MAX_ATTEMPTS - 1:
                time.sleep(2 ** attempt + random.random())   # backoff with jitter
                continue
            raise   # a 400 is your request, a 401 is your key

        if resp.stop_reason == "max_tokens":
            max_tokens *= 2   # the same request with more room, not a repair turn
            continue
        if resp.stop_reason != "end_turn":
            raise RuntimeError(f"no usable answer: {resp.stop_reason}")   # a refusal is never resent as is

        reply = text_of(resp)
        try:
            ticket = Ticket.model_validate_json(reply)   # its ValidationError is a ValueError
            check(ticket, source)
            return ticket
        except ValueError as err:
            log.warning("attempt %s rejected: %s", attempt, err, extra={"raw": reply})
            messages = messages + [   # append only, so the earlier prefix stays identical
                {"role": "assistant", "content": resp.content},   # the whole turn, thinking blocks included
                {"role": "user", "content": f"That ticket failed a check:\n{err}\nReturn only the corrected ticket."},
            ]
    raise RuntimeError(f"no valid ticket in {MAX_ATTEMPTS} attempts")

The parameter spellings come from the vendor's docs as of October 2026 and were not run here, so test against your SDK version first. Now the loop on a ticket that says "Charged twice in March, give this to dave."

The failure this prevents. Say the loop body also sends the email a ticket asks for, with no idempotency key, and a later line in the same try block, a write to the audit table, fails. Each pass gets a valid ticket, sends the email, and trips on the write. Three attempts send three emails to one customer, and the log shows three ordinary retries. The model call was safe to repeat; the send was not. A key per logical request, checked before sending, would have made attempts two and three do nothing.

When not to use it. A prototype needs no ladder; three attempts and a raise is the whole feature. Skip the repair loop where the call has side effects and you have no keys to make a second pass harmless. Skip it where latency is tight: every repair is another full call, so a user-facing path can double its wait. And do not use it to hide a prompt that fails often. If more than a few percent of calls need a repair, read the logged replies and fix the prompt, as in Versioning and testing prompts.

Next: those were failures your own system caused; Prompt injection is about the ones that arrive on purpose.