A/B testing and shipping a change safely

Everything from Regression testing a prompt or model change onward has been about deciding whether to merge. This lesson is about what happens after you do - because a golden set and a judge, however well calibrated, are still a proxy for the thing you actually care about: what happens when real people use the real thing.

1An eval score is a hypothesis

A judge's verdict, a regression run that stayed green, a pairwise comparison that favored the new version - all of it is evidence gathered against a fixed set of cases, scored by a fixed rubric. That's genuinely useful, and it's also not the same measurement as "users got more value from this." The golden set can't include a case it never anticipated. The judge's rubric can reward something - concision, a particular tone, a format - that doesn't actually move whether a real task got completed. None of that means the eval was worthless. It means a passing eval is the reason to try the change on real traffic, not the reason to skip trying it.

This is the same trap Why "looks good to me" fails opened the whole course with, one level up: eyeballing five demo cases wasn't enough, so you built a golden set - and now the golden set itself needs checking against the thing it's a proxy for. The fix is the same shape every time: don't trust the proxy alone, go measure the real thing, on a scale small enough that being wrong is cheap.

None of this means the earlier work was wasted. The regression gate from the last lesson still did its job - it caught the version that would have broken three existing cases outright, before a single user saw it. What it can't do is tell you whether the version that cleared the gate is actually the one people prefer. Those are two different questions, and shipping safely means answering both, not stopping at the first one because it's the one you already had a number for.

2Shadow, then canary, then a real comparison

The three stages differ in exactly one thing: how much of a real user's outcome depends on the new version before you've had a chance to notice a problem.

Run them in that order, not because the earlier stages are formalities, but because each one is cheaper to be wrong in than the next. Shadow traffic costs you compute and tells you if the new version is obviously broken - crashes, timeouts, wildly different output shape - with zero user impact. Canary traffic costs a small, bounded number of real users something if you're wrong, in exchange for a signal shadow mode structurally can't give you: how the new version performs when someone's actual request depends on the answer. Only once canary looks clean do you commit to a full comparison sized to actually settle the question.

The eval told you it should win. The rollout tells you if it did.

3Deciding it worked, before you start

Two decisions belong before you route a single real user to the new version, not after you're already looking at numbers and wondering what they mean.

Pick the real metric first. Not the judge's score again - something closer to what the user actually experienced: task completion, whether they asked a follow-up because the first answer didn't land, a regeneration or a thumbs-down, whether they abandoned the flow. The eval score got you here; it doesn't get to also be the metric that decides whether the rollout worked, or you've just measured your proxy against itself.

Set the rollback trigger before you ramp. Decide, in writing, what would make you pull the change back - an error rate above some level, a latency regression, the success metric dropping, cost per request spiking - and wire the check to act on it rather than waiting for someone to notice and raise an alarm. Deciding this while you're already staring at a dashboard that looks a little off is slower and more political than deciding it in advance, when nobody has a stake in the answer yet.

Keep the judge and the golden set frozen for the length of the test, the same discipline as the regression gate. If you're also running an automated judge over sampled live comparisons during the rollout, changing its model or rubric mid-test means you can no longer tell whether a metric moved because the candidate got better, or because the thing measuring it did - the exact failure How to trust a judge spent a whole lesson on preventing.

import hashlib

CANARY_PERCENT = 5          # start small - see the table above
ROLLBACK_THRESHOLDS = {
    "error_rate": 0.02,      # 2% - decided before the rollout started, not during it
    "p95_latency_ms": 4000,
}

def in_canary(user_id: str, percent: int = CANARY_PERCENT) -> bool:
    """Deterministic bucket: the same user always lands on the same side,
    so their experience doesn't flip between requests."""
    digest = hashlib.sha256(user_id.encode()).hexdigest()
    bucket = int(digest, 16) % 100
    return bucket < percent

def should_roll_back(metrics: dict) -> bool:
    return (metrics["error_rate"] > ROLLBACK_THRESHOLDS["error_rate"]
            or metrics["p95_latency_ms"] > ROLLBACK_THRESHOLDS["p95_latency_ms"])

# routing decision for one request
version = "candidate" if in_canary(user_id="user_48213") else "control"

# a scheduled check against live metrics - not a one-time glance
live_metrics = {"error_rate": 0.031, "p95_latency_ms": 2100}
if should_roll_back(live_metrics):
    print("rolling back: error_rate exceeded threshold")   # act automatically, don't wait

One more thing worth saying plainly: a canary on a couple hundred users that shows the new version slightly ahead is not a result yet, it's a coin that's landed heads a few extra times. How big a sample you need before you trust a difference is genuine statistics - sample sizing, confidence intervals, the mechanics of why a small early lead often evaporates - and it's worth reading up on properly rather than eyeballing a dashboard, because the mistakes here are well studied and avoidable. Full A/B is the stage sized to actually answer the question; canary is sized to catch a disaster, not to declare a winner.

None of this protects you from a different problem: the model behind an unchanged prompt quietly changing on its own, with no rollout of yours to blame. Drift: when the model changes under you picks up exactly that case.