Offline evals and online evals

1Two checks, at two different moments

Offline and online evals are not two competing ways of measuring the same thing. They run at different points in a change's life, against different data, and they exist to catch different kinds of failure. Neither one is optional.

The offline set is the gate: nothing ships past it without passing. The online sample is the smoke detector: it doesn't stop anything, it tells you something is burning while it's still small enough to matter. You need the gate because you don't want to find out about a known failure from a user. You need the smoke detector because the gate can only test what someone thought to put behind it.

Neither one substitutes for the other, and teams tend to over-invest in whichever one they built first. A team that only runs the offline set gets exactly the failure this lesson opened with - a clean pass on paper, a real problem live, and no mechanism watching for it. A team that only watches production, with no gate before shipping, finds out about every regression from a user instead of from a test run, which is the more expensive way to learn the same lesson. The two together give you what neither gives you alone: a check you control the timing of, and a check that's actually looking at what's happening right now.

2Why an offline win can still lose live

An offline pass is a claim about a specific, finite set of cases. It is not a claim about your users. Three gaps explain most of the disagreements you'll see between the two:

Coverage. The golden set was built from past traffic and past incidents. A new usage pattern - a different language, a much longer conversation, a feature that just launched - has, by definition, no cases in it yet. The eval isn't wrong about what it tested. It never tested the thing that broke.

Distribution. Even when the golden set includes a category, it may not include it in the proportion real traffic does. A prompt change that trades a rare failure for a common one can look neutral or even positive on fifty balanced cases and be clearly worse the moment ten thousand real requests, weighted toward the common case, hit it.

What's actually being measured. An offline score - exact match, a rubric, a judge's rating - is a proxy for "the user was satisfied." Sometimes the proxy and the real thing point in different directions: a rewritten answer might score higher against a rubric for completeness while reading, to an actual user, as longer and more annoying.

An offline win is a hypothesis about quality. Production is the experiment that checks it.

None of this means the offline set was a waste of effort. It means the offline set answers "did I break anything I already know about," and only production traffic can answer "does this actually work for everyone using it right now."

Treat a disagreement between the two as information rather than a tie-break to argue over. If offline says better and users say worse, something about the golden set or its grading is measuring the wrong thing, and that's worth finding before you trust the next offline run either.

3Running both, as a loop, not a one-off

The two checks work together as a loop, not as a sequence you complete once. Before any change - a prompt edit, a model swap, a new tool the system can call - run the golden set and require it to pass. After the change ships, keep sampling real traffic on an ongoing basis and scoring what comes back, at whatever rate you can afford to review.

The loop closes exactly the way the last lesson described: whatever online sampling catches becomes a new permanent case in the offline set. That's what keeps the gate from going stale. A golden set built once and never updated slowly turns into a test of last year's traffic.

import random

SAMPLE_RATE = 0.02  # review roughly 2% of production traffic; tune to your review capacity

def maybe_queue_for_review(request_id: str, input_text: str, output_text: str) -> None:
    """Call this after every real response. Most calls do nothing -
    a small, random slice gets written somewhere a human or a judge will look at it."""
    if random.random() < SAMPLE_RATE:
        review_queue.append({
            "request_id": request_id,
            "input": input_text,
            "output": output_text,
            "sampled_at": now_iso(),
        })
        # A separate, scheduled job scores this queue and flags anything
        # that looks wrong for a human to confirm - see the next part of this course.

Two percent of a busy system is still a lot of real cases every day, and unlike your golden set, it's a sample of exactly what's happening right now. When something in that queue turns out to be a genuine failure, it doesn't stay in the queue - it goes into the golden set, and the next offline run is checked against it forever.

The sample rate itself is a knob, not a constant. Turn it up right after a risky change - a new model version, a rewritten system prompt, a feature that touches a lot of traffic - when you most want an early read on whether something's off. Turn it back down once the change has settled and nothing unusual has shown up. What you're buying with that percentage is review capacity spent where it's most likely to find something, not an attempt to look at everything.

That's the whole loop: offline stops you from shipping a known mistake, online tells you about the mistakes nobody knew to look for, and every real failure moves permanently from the second category into the first.