Measuring retrieval

1The eval set is the work

Everything in this lesson depends on one artefact: a list of queries paired with the chunk ids that should come back for them. Without it, every metric below is undefined and every change you make is a guess.

Build it like this.

Take fifty real queries. Not fifty you invented at your desk - those are queries phrased the way you already think about the corpus, which is exactly the bias you are trying to measure past. Pull them from support tickets, search logs, your chat app's history, or ask five colleagues what they would type. Include the awkward ones: acronyms, typos, follow-ups, questions your docs do not actually answer.

For each, find the chunk ids that answer it. Search your own corpus, read the results, and write down the ids. Some queries have one right chunk, some have three, some have none - record the none ones too, they are how you test whether your app correctly says it does not know.

This part is manual. You can have a model draft candidates - point it at a chunk and ask what question this passage answers - and that is a reasonable way to get to a first draft fast. But if you ship the labels unchecked, you are measuring your retriever against another model's opinion, and any systematic mistake in that opinion becomes invisible to you forever. Read them. It is a day.

Fifty is enough to see a real change and small enough to actually finish. Grow it later, mostly by adding queries that failed in production - a failure you have written down is a regression test.

Fifty labelled queries is a day of work. Not having them costs you that day every month, in arguments about whether the answers feel worse.

2Three metrics, plainly

Set the eval up so it runs on retrieval alone - queries in, ranked chunk ids out, no generation call. It should take seconds and cost nothing.

recall@k is the share of queries whose correct chunk appears anywhere in the top k. It matters most because it is a hard ceiling: if the answer is not in the k chunks you paste into the prompt, the model cannot produce it. Recall@5 of 0.6 means four questions in ten are unanswerable before the model is called. No prompt fixes those.

MRR - mean reciprocal rank - averages 1 / rank of the first relevant chunk across queries. First place scores 1.0, third place scores 0.33, missing scores 0. It tells you about ordering, which is the thing reranking from lesson 08 is supposed to improve. If recall@50 is high but MRR is low, you have a ranking problem and a reranker is the fix. If recall@50 is low, ranking is not your problem.

nDCG@k - normalised discounted cumulative gain - is for when relevance is graded rather than binary. Some chunks fully answer the question, some give useful context, some are noise. nDCG assigns each a gain, discounts it by how far down it appeared, and divides by the best possible ordering so the number lands between 0 and 1. It is the standard metric in the IR literature and in benchmarks like BEIR, so you will meet it. It also needs graded labels, which is more annotation work. Binary labels and recall@k plus MRR will serve most teams fine.

# gold: query -> the chunk ids a human said actually answer it
gold = {
    "refund window for ebooks": {"policy-14"},
    "return an unopened blender": {"policy-15"},
    "who approves expenses over 5k": {"finance-3", "finance-4"},
}

# runs: what your retriever returned, best first
runs = {
    "refund window for ebooks": ["policy-14", "policy-2", "policy-9"],
    "return an unopened blender": ["policy-2", "policy-9", "policy-15"],
    "who approves expenses over 5k": ["hr-1", "finance-9", "finance-3"],
}

def recall_at_k(k):
    hits = [any(d in gold[q] for d in runs[q][:k]) for q in gold]
    return sum(hits) / len(hits)

def mrr():
    total = 0.0
    for q in gold:
        for position, doc_id in enumerate(runs[q], start=1):
            if doc_id in gold[q]:
                total += 1 / position
                break
    return total / len(gold)

print(f"recall@1 {recall_at_k(1):.2f}")   # 0.33
print(f"recall@3 {recall_at_k(3):.2f}")   # 1.00
print(f"MRR      {mrr():.2f}")            # 0.56

Forty lines and no dependencies. That is the whole barrier between you and knowing whether Tuesday's change helped.

Which k

Measure at the k you actually send to the model, and at the k your first stage returns.

The gap between those first two numbers is your reranker's opportunity. If recall@50 is 0.94 and recall@5 is 0.71, there are twenty-three questions in a hundred where the answer is in the candidate pile and not in the prompt. That gap is exactly what lesson 08 is for.

3Test the halves separately

This is the habit worth more than any individual metric.

A RAG pipeline has two failure modes, and end-to-end answer quality mixes them into one unusable number. Score them apart.

For the generation half, hand the model the chunks you know are correct - the gold ids from your eval set, not what retrieval returned. Now you are measuring the answering step in isolation. If it is wrong with the right passages in front of it, the prompt is the problem. If it is right there but wrong in production, retrieval is the problem. The build-with-llms lesson on evals covers scoring the generated text itself.

A worked diagnosis:

Running it

Wire the eval to a single command and run it on every change to chunking, the embedding model, the index, filters, fusion weights, the reranker or the candidate count. Every one of those moves retrieval, and every one of them is the kind of change that feels obviously good and sometimes is not.

Keep the number in the repo next to the code, one row per run, so the history is visible. When someone says the answers feel worse, you open the file.

Two honest limits. Fifty queries is a small sample - a change worth two or three questions is inside the noise, so do not celebrate small moves. And your eval set drifts as your corpus and your users change; add the queries that failed in production and re-read the labels every few months.