Reranking

1.Two stages, on purpose

Most search systems are built in two stages, and they have opposite jobs.

The reranker can only reorder what it is given. If the answer chunk was not among the fifty candidates, reranking does nothing at all. Stage one sets your ceiling, which is why Hybrid search comes before this lesson, not after.

Retrieval decides what is possible. Reranking decides what is on top. You need both, in that order.

Here is the arithmetic on an invented eval set. Take 100 questions and 50 candidates each. The answer chunk is in the first-stage top 5 for 62 questions. It is somewhere in the top 50 for 94. Rerank and keep five, and the answer reaches the model for 85 questions. The reranker moved 23 questions into the window. The other 6 were never candidates, so no reranker could have saved them.

Those 6 are the failure case you will meet most: the answer was never a candidate, and the reranker is blamed for it. Check recall@50 before you tune anything on stage two.

2.Why the second pass sees more

A single-vector bi-encoder - what your vector index uses - embeds the query and embeds the document separately, at different times. The document was embedded weeks ago at index time; the query is embedded now. Two vectors go into a similarity function and a number comes out. At no point does any word of the query see any word of the document, and the whole match has to squeeze through one fixed-size vector per side.

That constraint is what makes vector search fast. You precompute every document vector once. It is also what makes it imprecise: a passage on the right topic in the right register lands near the query whether or not it answers the question.

A cross-encoder takes the query and one passage, joins them into a single input, and runs them through the model together. Attention now runs across both. The words "how long" in the query can look at "30 days" in the passage. The model can notice that a passage names the topic but answers a different question or states the opposite. A bi-encoder is blind to that class of mistake.

A third shape sits between the two: late interaction, where token vectors are precomputed and meet the query only at scoring time. Late interaction: one vector per token covers it in full.

# pip install sentence-transformers
from sentence_transformers import CrossEncoder

# A small teaching model, as of October 2026 - not a recommendation.
model = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

query = "how long do I have to return a laptop"
candidates = [
    "Our returns desk is open on weekdays from 9am until 5pm.",
    "Electronics may be returned within 30 days of delivery.",
    "Laptop batteries are covered by a two year warranty.",
]

# One pass per pair. Scores are raw numbers: compare them
# within this one query, never across queries.
scores = model.predict([(query, text) for text in candidates])

for score, text in sorted(zip(scores, candidates), reverse=True):
    print(f"{score:6.2f}  {text}")

All three candidates are about laptops and returns, so their embeddings sit close together and vector search could order them any way. The cross-encoder reads "how long" against each one, and only the middle passage answers it.

How much can a reranker add? In ViDoRe V3, a benchmark paper from January 2026 (revised April 2026), a text cross-encoder reranker lifted one strong text retriever from about 50 to about 64 on nDCG@10, a score for the whole ordering (defined in Measuring retrieval), a gain of 13.2. A reranker built for page images, put over an image retriever, added almost nothing and hurt four datasets. So test the one you plan to use; a good reranker for one kind of input is not a good reranker for all. In "When to Use Graphs in RAG" (GraphRAG-Bench, June 2025), one setup with a 2023-era open embedder and reranker gained about 22 points of fact-lookup recall from the reranker, more than any graph system added.

3.What it costs, and how much to send

Reranking is not free, and vendor pages rarely say so. The cost is one scoring pass per candidate, on every query. Take 50 chunks of about 300 tokens each: the model reads roughly 15,000 tokens per query, where embedding the query read about 20. Double the candidates and compute roughly doubles, and for self-hosted or LLM rerankers so do latency and cost. Hosted APIs bill in their own unit, so check the pricing page for how candidates are counted.

No latency figure here: it moves with hardware, batching and chunk length. Measure p50 and p95 for your own candidate count on your own setup, and set the budget from the p95.

LLM-as-reranker means prompting a general model to score or order the candidates. It handles criteria a trained reranker never saw, such as "prefer the current policy version", but it is the most expensive option per query by a wide margin and adds a generation call to your latency budget. Use it offline, for evaluation, or when the criterion cannot be expressed any other way.

Two practical notes. Scores are for ordering, not for truth: Cohere's "Reranking best practices" page (as of October 2026) says its scores depend on the query, so a cutoff needs calibrating on 30 to 50 of your own queries. And an inference, not a tested result: if you check permissions live at the source, as in Permissions that travel with the data, run that check after the rerank on the top few, not on all fifty.

Choosing the candidate count

Call it n for candidates in, k for chunks out. n is the dial.

Compute recall@n for your first stage across the eval set from Measuring retrieval. Where recall@n stops climbing, extra candidates only add cost: if recall@50 were 0.94 and recall@200 were 0.95 (invented), you would pay four times as much for one extra question in a hundred. Judge stage one on recall at your n and leave ordering to stage two; Measuring retrieval explains why the metric must match the stage.

k is set by your prompt, usually 3 to 10, and not by recall. Chroma's July 2025 report on long inputs found that even one near-miss passage lowered answer accuracy when the right one was present. It tested older models and no RAG pipeline, so take the direction and not a number: every weak chunk you add is a distractor. Pick k by answer quality, as Measuring the whole answer shows.

Where it pays most, and when not to use it

It pays most on near-duplicate corpora (twelve versions of one policy, or one answer restated across many support articles), where first-stage scores land within a hair of each other and the order is close to arbitrary. It also helps on ambiguous short queries, on long documents whose chunk embeddings look alike, and after hybrid search, whose fused list has good recall and rough order.

Do not reach for it when the corpus is small and its documents are short and clearly distinct, when queries are single identifiers that BM25 already ranks first, or when your latency budget has no room for fifty extra passes. Do not reach for it when recall@n is the problem either: that is a stage-one fix, such as hybrid search, or Query rewriting. Reranking sharpens a good candidate list; it cannot repair a bad one.