Reranking

1Two stages, on purpose

Every serious search system is built in two stages, and they have opposite jobs.

Read the last row twice. The reranker can only reorder what it is given. If the answer chunk was not in the fifty candidates, reranking does nothing at all. Stage one is still the one that decides your ceiling - which is why lesson 07's hybrid search comes before this lesson, not after.

Retrieval decides what is possible. Reranking decides what is on top. You need both, in that order.

2Why the second pass sees more

This is the mechanism, and once you see it the rest of the lesson is obvious.

A bi-encoder - what your vector index uses - embeds the query and embeds the document, separately, at different times. The document was embedded weeks ago at index time; the query gets embedded now. Two vectors go into a similarity function and a number comes out. At no point does any word of the query see any word of the document. The whole match has to be squeezed through a single fixed-size vector per side.

That constraint is what makes vector search fast. You precompute every document vector once, and at query time you compare numbers. It is also what makes it imprecise: a passage that discusses the right topic in the right register produces a vector near the query's whether or not it answers the question.

A cross-encoder takes the query and one passage, glues them into a single input, and runs them through the model together. Now attention runs across both. The word "fourteen" in the passage can look at the word "how long" in the query. The model can notice that the passage mentions the topic but answers a different question, or that it states the opposite, or that it is a table of contents entry that merely names the topic. That is the class of mistake a bi-encoder is structurally blind to.

Which explains the shape. You cannot run a cross-encoder over a million chunks per query - that is a million model passes. You can run it over fifty. So you use the cheap method to get fifty plausible candidates and the expensive method to order them.

from sentence_transformers import CrossEncoder

# Reads the query and each passage together, one pass per pair.
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L-6-v2")

query = "how long do I have to return a laptop"
candidates = [
    "Our returns desk is open on weekdays from 9am until 5pm.",
    "Electronics may be returned within 30 days of delivery.",
    "Laptop batteries are covered by a two year warranty.",
]

scores = model.predict([(query, text) for text in candidates])

for score, text in sorted(zip(scores, candidates), reverse=True):
    print(f"{score:6.2f}  {text}")

All three candidates are about laptops and returns, so their embeddings sit close together and vector search could order them any way at all. The cross-encoder reads "how long" against each one and only the middle passage answers it.

3What it costs, and how much to send it

Reranking is not free and the marketing rarely says so. The cost is one scoring pass per candidate, on every query. Fifty candidates is fifty passes. Double the candidates and you double both the latency and the bill for that stage.

The hosted rerankers - Cohere's Rerank, Voyage's, Jina's - are the usual starting point because they are one API call and no infrastructure. Prices and model names move, so look them up rather than trusting a number in any tutorial, including this one.

LLM-as-reranker means prompting a general model to score or order the candidates. It is worth knowing about because it handles criteria a trained reranker never saw - "prefer passages from the current policy version", "prefer ones that give a number". It is also the most expensive option per query by a wide margin, and it adds a generation call to a latency budget that already has one. Use it for offline work, for evaluation, or when the criterion genuinely cannot be expressed any other way.

Choosing the candidate count

Call it n for candidates in, k for chunks out. k is set by your prompt budget, usually 3 to 10. n is the dial.

The right way to pick it is measurement, not a rule. Compute recall@n for your first stage across your eval set from lesson 10. The point where recall@n stops climbing is the point where extra candidates only add cost - if recall@50 is 0.94 and recall@200 is 0.95, you are paying four times as much for one extra question in a hundred.

Where it pays most

• Long or verbose documents. Chunk embeddings average away detail, so many chunks look alike. The reranker reads the actual text.
• Ambiguous queries. Short questions produce weak, undifferentiated similarity signals.
• Near-duplicate corpora. Twelve versions of the same policy, or the same answer restated across many support articles. First-stage scores land within a hair of each other, so the ordering is close to arbitrary. This is where reranking helps most.
• After hybrid search. Fusion from lesson 07 gives you a merged candidate list with good recall and rough ordering. The reranker is exactly the tool for fixing rough ordering.

Where it helps least: small corpora of short, clearly distinct documents, and single-identifier lookups where BM25 already put the right answer first.