Measuring retrieval

1.The eval set is the work

Everything here depends on one artefact: queries paired with the chunk ids that should come back for them. Without it every metric is undefined and every change is a guess. Choosing an embedding model and Chunking used thirty labelled queries for one decision each. Treat that set as the seed and grow it to fifty, since this one must judge every change from now on.

Build it like this.

Take fifty real queries. Not fifty you invented at your desk - those are phrased the way you already think about the corpus. Pull them from support tickets, search logs or chat history, or ask five colleagues what they would type. The Evals course's Building an eval set from your own traffic goes deeper. Include the awkward ones: acronyms, typos, follow-ups, questions your docs do not answer.

For each, find the chunk ids that answer it. Some queries have one right chunk, some have three, some have none. Record the none ones too, since they test whether your app says it does not know. If an answer needs several documents, keep two label levels: evidence documents that support each step, and gold documents that hold the answer. BrowseComp-Plus, an August 2025 benchmark paper, labels both, so you can score retrieval alone and the whole run.

This part is manual. A model can draft candidates - show it a chunk and ask what question it answers. But unchecked labels mean you are measuring against another model's opinion, and its systematic mistakes stay invisible. Read them. It is a day.

Now the failure that catches careful teams. If you find answers by reading your current retriever's top results, the gold set holds only what it already finds. Swap in a better retriever and it returns a right chunk nobody labelled, and the score calls it a miss. BrowseComp-Plus saw this: with extra unjudged documents added, dense retrievers scored lower, since unjudged documents count as non-relevant, though some may be useful. So label from the merged results of two different retrievers, such as BM25 and vectors, and add any new miss that reads like a good answer. Snapshot the corpus with the labels, as that benchmark does, so a score change is the retriever's and not the data's. Grow the set with queries that failed in production - a written-down failure is a regression test.

Fifty labelled queries is a day of work. Not having them costs you the same day again, in arguments about whether the answers feel worse.

2.Four metrics, plainly

Run the eval on retrieval alone - queries in, ranked chunk ids out, no generation call. It takes seconds and costs next to nothing.

recall@k is the share of queries with a right chunk anywhere in the top k. It is a hard ceiling: if the answer is not in the k chunks you paste into the prompt, the model cannot produce it. Recall@5 of 0.6 means four questions in ten are unanswerable before the model is called. This lesson counts a query as a hit when any one of its gold chunks appears; if an answer needs all of them, divide found by needed instead.

MRR - mean reciprocal rank - averages 1 / rank of the first relevant chunk across queries. First place scores 1.0, third place 0.33, missing 0. It measures ordering, which Reranking is meant to improve: recall@50 high but MRR low is a ranking problem; recall@50 low is not.

nDCG@k - normalised discounted cumulative gain - is for graded relevance. Each result's grade is divided by log2(rank + 1), summed, then divided by the best possible ordering. By hand: grades 3 (full answer), 2 (useful), 0 (noise) come back in the order 3, 0, 2. That scores 3/1 + 0/1.58 + 2/2 = 4.0. The best order, 3, 2, 0, scores 3 + 2/1.58 = 4.26, so nDCG@3 is 4.0 / 4.26 = 0.94. Libraries differ on the gain formula, so check yours. Graded labels cost more annotation; binary labels with recall@k and MRR serve most teams.

# gold: query -> the chunk ids a human said actually answer it
gold = {
    "refund window for ebooks": {"policy-14"},
    "return an unopened blender": {"policy-15"},
    "who approves expenses over 5k": {"finance-3", "finance-4"},
}

# runs: what your retriever returned, best first
runs = {
    "refund window for ebooks": ["policy-14", "policy-2", "policy-9"],
    "return an unopened blender": ["policy-2", "policy-9", "policy-15"],
    "who approves expenses over 5k": ["hr-1", "finance-9", "finance-3"],
}

def recall_at_k(k):  # a hit if any gold chunk is in the top k
    hits = [any(d in gold[q] for d in runs[q][:k]) for q in gold]
    return sum(hits) / len(hits)

def mrr():
    total = 0.0
    for q in gold:
        for position, doc_id in enumerate(runs[q], start=1):
            if doc_id in gold[q]:
                total += 1 / position
                break
    return total / len(gold)

print(f"recall@1 {recall_at_k(1):.2f}")   # 0.33
print(f"recall@3 {recall_at_k(3):.2f}")   # 1.00
print(f"MRR      {mrr():.2f}")            # 0.56

About thirty lines and no dependencies - that is the barrier between you and knowing whether Tuesday's change helped.

Which k

Measure at the k you send the model and at the k your first stage returns. The gap is your reranker's opportunity: with invented numbers, recall@50 of 0.94 against recall@5 of 0.71 leaves twenty-three questions in a hundred with the answer in the pile and not in the prompt.

The metric must match the stage. An August 2026 preprint compared eight chunking methods on three small open embedding models (dense search only). Enriched chunks often ordered the top ten best on nDCG@10, while plain token windows were competitive on Recall@100. The best method changed with the metric. Judge a first stage on recall at its candidate count, and the final order on MRR or nDCG.

Recall can also hide padding. GraphRAG-Bench (preprint, June 2025, revised February 2026) reported graph systems with high evidence recall but low context relevance. Watch precision@k beside recall@k.

3.Test the halves separately

This habit is worth more than any single metric. A RAG pipeline fails in two places, and end-to-end quality blurs them into one number. Score retrieval first, as above. Then test generation separately, with the gold chunks as input; Measuring the whole answer shows how. If both halves are weak, fix retrieval first.

Wire the retrieval eval to one command and run it on every change to chunking, the embedding model, the index, filters, fusion weights, the reranker or the candidate count. Keep one row per run in the repo, so when someone says the answers feel worse, you open the file.

Fifty queries is a small sample: each query is two points of recall, so a change worth two or three questions is inside the noise. Public benchmarks have the same problem. The ColBERT-Zero preprint (February 2026) compared two training recipes 0.31 nDCG@10 points apart on BEIR (a public retrieval benchmark, 2021) and cites "the potential for noise from BEIR". Measure on your own set rather than trusting a fraction of a point on someone else's.

When not to use it. With fewer than about thirty queries, read the results by hand instead of computing a score. Where the answer is not inside a chunk - counting rows, summarising a report - recall@k measures nothing; see When RAG is the wrong tool. And when an agent issues many searches per question, the first query's top k understates what it saw, so score the evidence it collected across the run.

The set drifts as corpus and users change, so re-read the labels every few months. A good retrieval score is still only a ceiling, and Measuring the whole answer checks what the model does with it.