Why "looks good to me" fails

1Five examples is an anecdote, not a sample

Here is what actually happens when you "try a prompt change and see how it looks." You open a scratch file, or the vendor's playground, and you type in a handful of inputs. Which inputs? The ones you already have on hand - last week's support ticket, the query from the demo, the one case a teammate mentioned. You run the old prompt, you run the new one, and you compare.

Two things go wrong before you have read a single output.

The sample is tiny. Five or ten cases is not a small sample of your traffic, it is close to no sample at all. If one input in twenty trips up the new prompt, you need on the order of dozens of tries just to have a even chance of hitting it once, and you are not running dozens, you are running the five you already had.

The sample is not random - you chose it, and you chose it badly for this purpose. You reach for cases you already understand, because those are the ones you can judge quickly. That means you systematically under-sample the inputs that are long, or ambiguous, or in a register you do not personally write in. Those are exactly the inputs most likely to break something, and they are the ones least likely to appear in a five-minute manual check.

Put those two together and you get a check that is biased toward telling you the change is fine. It is not lying to you. It is answering a much narrower question than the one you think you asked.

There is a third failure hiding in the first two, and it's the quiet one: you also decide when to stop. The moment the five examples look better, you stop trying inputs. You do not keep going looking for the sixth case that breaks it, because the point of the exercise, in your head, was to confirm the change worked - not to find out whether it didn't. That stopping rule is doing as much damage as the small sample itself.

2The benchmark score is the same trap in a nicer outfit

It is tempting to think a public benchmark solves the sample-size problem, because the number is big and somebody else computed it carefully. It does not, and the reason is the same reason the five-example demo fails: the sample was chosen by someone who is not you, for a purpose that is not yours.

A benchmark is a fixed set of questions, written by researchers, published once, and then reused by everyone comparing models against it for years. That creates two problems that have nothing to do with how careful the benchmark authors were. First, the questions are general by design - they test broad reasoning or broad knowledge, not the specific shape of the requests your product actually receives, your domain's vocabulary, or the tone your users write in. Second, a benchmark that has been public for a while has a real chance of having leaked into the training data of the very models being scored on it, which quietly inflates the number without improving anything you would notice in production.

Neither problem is a scandal. It is just what a benchmark is: someone else's narrow sample, standing in for the much wider range of things your system has to get right.

A benchmark score is somebody else's five examples, run once, with a leaderboard attached.

None of this means benchmarks are worthless - they are a reasonable first filter when you are choosing between models you have never used, before you have any traffic of your own to test with. The mistake is treating a leaderboard position as evidence about your product, months into building it, instead of treating it as a rough starting point you replace with your own numbers as soon as you can.

3What actually tells you something

The fix is not a bigger demo, and it is not a better leaderboard. It is a fixed set of cases that came from your own traffic, checked the same way every time you make a change, so that "looks good" turns into a number you can compare against the last number.

You do not need this to be sophisticated to get value out of it immediately. Even five real cases, checked by code instead of by eye, beat five real cases checked by memory - because a script does not get tired, does not selectively remember the ones that passed, and produces the same answer if you run it twice.

# The smallest possible upgrade from "looks good to me": stop trusting your memory,
# start trusting a record. This is not yet a real eval set - just five cases,
# saved once, so the *next* run is a comparison instead of a fresh impression.

cases = [
    {"input": "cancel my subscription", "expected_contains": "cancel"},
    {"input": "why was I charged twice", "expected_contains": "refund"},
    {"input": "asdkjhasd", "expected_contains": "didn't understand"},
    {"input": "wht is ur refund policy???", "expected_contains": "refund"},
    {"input": "", "expected_contains": "didn't understand"},
]

def run_case(prompt_fn, case):
    output = prompt_fn(case["input"])
    passed = case["expected_contains"].lower() in output.lower()
    return {"input": case["input"], "output": output, "passed": passed}

def run_all(prompt_fn):
    results = [run_case(prompt_fn, c) for c in cases]
    print(f"{sum(r['passed'] for r in results)}/{len(results)} passed")
    return results

Run this before a change and after it, and you already have something a demo never gives you: a count, and a record of exactly which case flipped. It is still only five cases, so it still cannot tell you much about the other five thousand - that gap is real, and closing it is the whole subject of the next lesson, Building an eval set from your own traffic. But it has already stopped being a vibe. That is the entire point of this lesson: the failure was never "not enough time to test." It was mistaking an impression for a measurement.