Hybrid search

1What embeddings are built to lose

Lesson 01 said an embedding is a point in space where nearby means similar in meaning. That is the whole value and it is also the whole problem.

To place "the refund window is fourteen days" near "how long do I have to get my money back", the model has to throw away the exact words. It keeps the gist. That is a feature when the user paraphrases, and a bug when the user types something whose gist is nothing.

Consider what a rare token actually is. ERR_CONN_RESET gets split by the tokeniser into fragments, those fragments get averaged into the meaning of the sentence around them, and what comes out is a vector sitting somewhere near "connection problem". So does every other chunk about connections. The chunk that literally names the error is in the neighbourhood, but it is not obviously the closest point in it.

Look at the last two rows. Real queries are usually a mix: some meaning, some exact strings. Neither system alone covers a mix.

Embeddings find text that means the same thing. Keyword search finds text that says the same thing. Most real questions need both.

2BM25, without the formula

The keyword half of hybrid search is almost always BM25, a scoring function that has been the default in Lucene, Elasticsearch, OpenSearch and Solr for years. You do not need the algebra. Three ideas carry it.

Rare terms count for more. If a query word appears in nine out of ten documents, matching it tells you nothing. If it appears in three documents out of a million, matching it tells you almost everything. BM25 weights each query term by how rare it is across the corpus. This is why it nails identifiers - an error code is about as rare as a term gets.

Repeats saturate. A document that says "refund" ten times is more about refunds than one that says it once. But not ten times more. BM25 lets term frequency flatten out quickly, so keyword stuffing stops paying after the first few occurrences.

Long documents are penalised. A ten-thousand-word page matches your query words by accident. BM25 divides by a length term, so matching in a short focused chunk is worth more than matching somewhere in a sprawling one.

That is it. No learning, no training, no embeddings. It is corpus statistics and it is a genuinely hard baseline to beat.

Note the last row. The two systems fail on opposite inputs, which is exactly why combining them works. If they failed on the same queries there would be no point.

3Merging two ranked lists

You run both searches. You get two lists back. Now what?

The obvious idea - add the scores - does not work, and it is worth being precise about why. A cosine similarity is bounded, and in practice most of your chunks cluster in a narrow band like 0.6 to 0.85. A BM25 score is unbounded and depends on corpus statistics; 14.2 is meaningful only relative to the other scores for that same query. There is no exchange rate between them. Min-max normalising each list per query looks like a fix but is unstable, because the range shifts query to query - one outlier drags the whole scale.

Reciprocal Rank Fusion dodges the problem by ignoring the scores entirely. For each system, a document at position r contributes 1 / (k + r), with k conventionally 60. Add up the contributions across systems and sort. That is the whole method.

The behaviour it gives you is exactly what you want. Being ranked highly by both systems beats being ranked first by one and missed by the other. Position 1 in one list scores 1/61 = 0.0164; position 5 in both lists scores 2/65 = 0.0308, nearly double. Agreement wins.

# Merge ranked lists with Reciprocal Rank Fusion.
# Only positions are used, so nothing needs normalising.

def rrf(rankings, k=60):
    scores = {}
    for ranking in rankings:
        for position, doc_id in enumerate(ranking, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + position)
    return sorted(scores.items(), key=lambda pair: -pair[1])

vector_hits = ["doc-9", "doc-2", "doc-7", "doc-4"]   # nearest neighbours
keyword_hits = ["doc-4", "doc-9", "doc-1", "doc-3"]  # BM25 on the same query

for doc_id, score in rrf([vector_hits, keyword_hits])[:3]:
    print(f"{doc_id}  {score:.4f}")

# doc-9  0.0325   ranked 1st and 2nd - both systems agree
# doc-4  0.0320   ranked 4th and 1st - close behind
# doc-2  0.0161   only one system found it

What k does: a larger k flattens the curve, so the gap between rank 1 and rank 10 shrinks and agreement across lists matters more. 60 is the value from the original paper and it is a fine place to start. Tune it only after you have the measurement from lesson 10.

Weighted score blending is the alternative: normalise both score sets and compute w * dense + (1 - w) * sparse. It can beat RRF when you have an eval set to tune w on and your score distributions are stable. It also breaks quietly when the corpus grows and the distributions shift. Start with RRF; move to weights only if measurement says you gained something.

What it costs

You may not need two systems. Postgres with pgvector alongside a tsvector column does both in one database. Elasticsearch and OpenSearch both index dense vectors next to their BM25 fields. Most hosted vector databases now expose a sparse or hybrid mode. Check what you already run before adding infrastructure.

Where it clearly pays

Hybrid is a clear win when your corpus is full of names that only appear in it: error codes, SKUs, ticket ids, internal service names, API method names, control numbers, acronyms your company invented. That describes most engineering documentation, most support corpora and most legal or compliance sets.

It helps less when the corpus is prose about general topics and users ask in general language - marketing pages, essays, a book. There, the vocabulary overlaps between users and documents anyway, and dense retrieval already handles it.

There is one honest caveat. Hybrid usually improves recall, and lesson 10 is where you check whether it improved yours. Do not take the improvement on faith because a blog post reported a number on a benchmark that is not your corpus.