How to trust a judge

Deterministic checks and LLM-as-judge told you when judgment genuinely needs a model instead of a regex. This lesson is about the model you just decided to trust. A judge is not a special kind of code that happens to be reliable because it's a language model - it's a language model, with the same failure modes as the thing it's grading. It can be confidently wrong. It has never had a human check its work unless you made that happen.

1Grade the judge before it grades you

Calibration means one thing: does the judge agree with a human on cases where you already know the right answer? You already have the raw material for this. The gold set from Building an eval set from your own traffic - real inputs, expert-written edge cases, the incidents you turned into permanent test cases - is also the set you use to score the judge itself. Have a human label a batch of outputs pass or fail, or better vs. worse. Run the same batch through the judge. Compare.

The comparison that matters is agreement, not "the judge got most of them right." Two graders can each be individually reasonable and still disagree with each other constantly, which is exactly what a raw percentage hides. Cohen's kappa is the standard tool here: it measures agreement between two raters while subtracting out the agreement you'd expect from chance alone. A kappa near zero means the judge is barely better than a coin flip next to your human; a kappa above roughly 0.6 is the conventional line for "trustworthy enough to lean on," though where exactly you set that bar depends on how much a wrong judgment costs you.

That last part matters more than the number. A judge that passes something a human would reject - a false positive - is the dangerous direction, because it creates confidence in output nobody actually checked. A judge that fails something a human would accept is annoying but safe: it just means a human looks at one more case than strictly necessary. If you're grading anything where a wrong pass reaches a user unsupervised, weight your calibration set and your threshold toward catching false positives specifically, not just toward a good overall kappa.

None of this needs to be elaborate to be worth doing. A hundred labeled cases and an afternoon spent reading the judge's verdicts next to a human's is enough to know whether you're looking at a genuine second opinion or a system that agrees with itself. The mistake is skipping the step entirely because the judge's scores looked plausible - plausible is not the same claim as calibrated, and only one of those two is checkable.

An unvalidated judge is not a second opinion. It's a guess that never admits doubt.

2The two biases that show up first

Once a judge is broadly calibrated, two specific failure modes account for most of the gap between its score and a human's. Both are structural - they come from how the judge reads the input, not from a bad rubric.

Position bias is the easiest to test and the easiest to miss, because a single run of your eval set never shows it to you - you'd need the same pair scored twice, in both orders, to see the verdict move. That's worth building into the harness itself rather than treating as a one-off audit.

from itertools import product

from anthropic import Anthropic
from sklearn.metrics import cohen_kappa_score

client = Anthropic()

def judge_pair(question: str, answer_a: str, answer_b: str) -> str:
    """Returns 'A' or 'B' - whichever the judge prefers."""
    resp = client.messages.create(
        model="claude-sonnet-5",
        max_tokens=5,
        system="Answer with exactly one letter: A or B. Judge on correctness only.",
        messages=[{"role": "user", "content":
            f"Question: {question}\n\nA: {answer_a}\n\nB: {answer_b}\n\nWhich is better?"}],
    )
    return resp.content[0].text.strip()[0]

def check_position_bias(cases: list[dict]) -> float:
    """Fraction of cases where swapping order flips the verdict."""
    flips = 0
    for case in cases:
        first = judge_pair(case["question"], case["answer_a"], case["answer_b"])
        second = judge_pair(case["question"], case["answer_b"], case["answer_a"])
        # second run swapped the slots, so a stable judge should give the opposite letter
        if first == second:
            flips += 1
    return flips / len(cases)

# Separately: how well does the judge track a human, on labeled cases?
human_labels = [1, 0, 1, 1, 0, 1, 0, 0]     # 1 = human said "pass"
judge_labels = [1, 0, 1, 0, 0, 1, 0, 1]     # judge's verdict on the same cases
print(f"agreement (kappa): {cohen_kappa_score(human_labels, judge_labels):.2f}")

A check_position_bias result well above zero means the judge is, at least sometimes, voting for a slot instead of an answer. A kappa printed at the bottom below your threshold means don't ship decisions on this judge's word yet - go fix the rubric or swap the judge model, then recalibrate.

3Keeping a judge honest over time

Calibration is not a box you tick once at launch. Two things change under a judge without you touching anything: the provider updates the model behind the API name, and your product drifts into cases your original gold set never covered. Both mean a judge that was trustworthy in January can be wrong by June with no error message telling you so.

The practical response is boring on purpose: freeze the judge's model version and settings for the duration of any single comparison you care about, so a score difference is about your change and not about the judge changing underneath it. Then recalibrate against the gold set on a schedule - monthly is a reasonable default - and again immediately after anything that might have shifted its behavior: a provider model update, a rubric edit, a big change in what your product handles. Alert on the kappa the same way you'd alert on any other metric that quietly slipping past a threshold; a judge you stopped checking is a judge you're trusting on faith, and faith is exactly what a gold set exists to replace.

Once the judge itself is something you trust, the next question is what to do with a human reviewer's time now that most of the grading doesn't need one - which is where Pairwise comparison and human review that scales picks up.