Validating what comes back

Structured outputs in practice ended with the decoder holding the shape and the SDK holding the range. Neither can hold the truth. This lesson is about everything still wrong after the parse succeeds, and none of the checks in it needs a second model call.

1.Three kinds of wrong, and why you must tell them apart

Engineers tend to keep one bucket labelled "the model failed". It is too coarse to act on: the three things in it want three different responses.

With a decoder-enforced schema the first row mostly shrinks, as of October 2026: parse errors become rare, and what is left is a reply that never finished. The other two rows do not shrink. The decoder guarantees a valid object, not a true one.

A retry is worth paying for only if something changed. For an unfinished reply it is the cap or the effort. For a fabricated one it is the error text you attach, which Retries and fallbacks covers. A well-formed wrong answer has nothing to attach, because no check knows it is wrong, so asking again returns the same misreading.

2.Validate at the boundary, exactly like an HTTP body

You would never write a JSON body from the internet to your database unchecked. A model reply has the same standing: it came over the network, its content is shaped by input you do not control, and it is right most of the time, which is the dangerous kind of wrong.

Three layers stand between the model and your row. The decoder holds the shape. A validator at the edge, Pydantic or Zod, holds types and ranges. Your code holds everything that depends on your data, and that is this lesson. The middle layer has one trap: the Python helper validates the reply against your model after the call, but the TypeScript plain-JSON-schema helper does not validate at all, and the Zod helper does (vendor docs, as of October 2026). Know which one you hold.

The checks that catch production bugs:

• Enum membership, after normalising. The set is closed, but the vendor documents that the case of a value can drift even under enforcement, as of October 2026. Compare case-insensitively, or normalise on purpose before you compare, never by accident.
• Present is not meaningful. An empty string passes a str check and says nothing. Decide whether empty is legal.
• Referenced ids that exist. This is the big one. The model returns owner: "dave" or product_id: "SKU-4471". No schema can know whether either exists. A set lookup or a query can, for almost nothing.
• Cross-field consistency. end_date after start_date. A status of resolved with a null resolution.

Tool-call arguments are model output too. If the model asks to run send_email, the arguments get the boundary check before the tool does anything. A 2025 benchmark of poisoned MCP tool descriptions, on older models with naive payloads, found the most effective attack was the quiet one: the user's real call still happened, with one value swapped, such as a recipient (the ranking is solid; the exact figures do not reproduce). A recipient allowlist in code stops that. Links and image addresses in a reply get the same treatment against a domain allowlist, since an address can carry data out; Containing prompt injection has the argument.

A schema proves the output has the right shape. Nothing in it can tell you the answer is true.

3.Grounding, a worked example, and what to keep

When the answer is supposed to come from a document you supplied, a ticket, a contract, a retrieved chunk, you have an unusually strong check: ask for the exact supporting span as a field, then search for it in the source. If it is not there, the reply is fabricated however well it reads.

Two things make it work. Compare on a normalised copy. Collapse whitespace and straighten smart quotes, or you will reject honest quotes. Give the model a legal way out. If the source does not hold the answer, it needs a permitted reply, null for the owner, say, or it will invent something to fill the field. That is the escape hatch from Output contracts, doing real work.

Here is the ticket from Structured outputs in practice, with the checks only your code can make. The parameter spellings are copied from the vendor docs as of October 2026 and were not run here, so try them on your SDK version first.

import logging
import re
from typing import Literal

from anthropic import Anthropic
from pydantic import BaseModel, Field

client = Anthropic()
log = logging.getLogger(__name__)
MODEL = "claude-sonnet-5-5"   # as of October 2026
OWNERS = {"alice", "bob", "priya"}   # facts the schema cannot know

def text_of(resp) -> str:
    return "".join(b.text for b in resp.content if b.type == "text")

class Ticket(BaseModel):
    category: Literal["billing", "bug", "feature_request", "other"]
    urgency: int = Field(ge=1, le=5)   # the SDK checks this after the call
    owner: str | None                  # required, with an explicit null
    quote: str                         # must appear in the source text

class BadReply(Exception):
    """The message is written to be read, by a person or by a retry."""

def squash(s: str) -> str:
    s = s.replace("‘", "'").replace("’", "'")
    s = s.replace("“", '"').replace("”", '"')
    return re.sub(r"\s+", " ", s).strip()

def problems(t: Ticket, source: str) -> list[str]:
    found = []
    if t.owner is not None and t.owner.casefold() not in OWNERS:
        found.append(f"unknown owner {t.owner!r}; use one of {sorted(OWNERS)} or null")
    if not t.quote.strip() or squash(t.quote) not in squash(source):
        found.append("quote does not appear in the ticket text")
    return found

def triage(text: str) -> Ticket:
    resp = client.messages.parse(
        model=MODEL,
        max_tokens=2000,   # thinking is on by default and counts against this
        system="Triage the ticket. Quote it exactly. Use null for an owner it does not name.",
        messages=[{"role": "user", "content": text}],
        output_format=Ticket,
    )
    # The schema holds only for a finished answer: a refusal or a cut-off may not match it.
    if resp.stop_reason != "end_turn":
        raise RuntimeError(f"no usable answer: {resp.stop_reason}")
    ticket = resp.parsed_output
    found = problems(ticket, text)
    if found:   # keep the raw reply and the input together, then fail loudly
        log.error("rejected %s | model=%s stop=%s | input=%r | raw=%r", found, MODEL, resp.stop_reason, text, text_of(resp))
        raise BadReply("; ".join(found))
    return ticket

Each failure carries a sentence that says what is wrong and what a good value looks like. That is deliberate: it is the text you would hand back to the model in a repair call.

The failure case. Picture a team that ships the schema and deletes the repair code. Weeks later someone spots tickets assigned to an engineer who left in the spring, and a summary quoting a sentence that is not in the thread. Every row was valid. The two checks above fix it, but the worse part was the log: they had kept the exception and thrown away the reply, and the input worked fine when re-run. Log the raw reply on every validation failure, next to the input, the stop_reason and the model id. Model failures do not reproduce reliably, and the reply you did not save is the only evidence there was. It can hold customer data, so decide how long you keep it; that last part is my advice, not the vendor's.

When not to use it. Grounding only works when the answer is a span of the source; for a summary or an inference there is no exact sentence to find. It also proves less than it feels like: a real quote can still back the wrong label, which is the well-formed-but-wrong row again, and only the prompt, the examples or a test set can fix that. Do not build a second-model judge before the free checks, and for a throwaway script a stop_reason check and a type check are enough.

Next: a reply that failed validation is now an exception with a readable message, and Stop reasons and refusals is about the replies that arrive with no body to validate at all.