Query rewriting

1The gap between the question and the corpus

Lesson 03 chunked your documents and lesson 02 picked the model that embeds them. Both of those improve the corpus side of the match. This lesson is the other side.

A retriever compares one string to a corpus. If that string is a poor description of what the user wants, no amount of index tuning helps. And user queries are poor descriptions surprisingly often, in four specific ways.

All four are the same underlying problem stated four ways: the string you searched with is not the string that would have found the answer. So change the string first.

Retrieval is only as good as the query it is given. If the search is failing, the query is a suspect, not just the index.

2The four rewrites

Expansion

Add terms the answer is likely to contain: synonyms, spelled-out acronyms, the formal term for a colloquial one, and near-neighbour vocabulary. PTO carryover becomes PTO paid time off annual leave holiday carryover rollover accrual.

This one helps the keyword half of hybrid search far more than the vector half - BM25 matches literal terms, so giving it more of the right terms directly raises its chances. On the dense side, expansion helps less and can hurt, because piling on words dilutes the embedding towards a general topic vector.

Cheap version, no model needed: a synonym dictionary and an acronym table for your domain. If you have fifty internal acronyms, a lookup table fixes fifty query classes for the price of a JSON file. Do that before you add a model call.

Decomposition

Split a multi-part question into one query per part, retrieve for each, then merge the results - RRF from lesson 07 works fine as the merge.

Use it when the question contains "and", "versus", "difference between", or two distinct nouns from different parts of your docs. The reason is mechanical: your chunks were split by topic, so a chunk covering both halves of a comparison usually does not exist. One search returns one side of it and the model answers half the question with total confidence.

The cost is more than latency. Three sub-queries at ten chunks each is thirty chunks in your prompt, so either you rerank down (lesson 08) or you pay for the context.

HyDE

HyDE stands for Hypothetical Document Embeddings. Have the model write a plausible answer to the question, then embed and search with that instead of the question.

The reason it works is a shape mismatch. Your corpus is made of statements. Your queries are questions. "Why is my build slow?" and "Build times increase when the dependency cache is invalidated on every run." are about the same thing, but as text they look quite different - different length, different grammar, different vocabulary. A hypothetical answer looks like a passage, so it lands nearer to real passages in the vector space.

The model does not need to be right. It needs to guess the right vocabulary and register. Even a factually wrong hypothetical answer usually pulls the query into the right neighbourhood, which is all retrieval needs.

HyDE costs a full generation call before you have searched anything, so it is the most expensive of the four. It pays on corpora where questions and passages are genuinely dissimilar - technical documentation, research papers, dense reference material. It pays much less on FAQ-shaped corpora, where the corpus is already made of questions.

Conversational rewriting

This is the one most chat apps get wrong, and it is the one with the clearest payoff.

In a conversation, meaning accumulates across turns. "What about the second one?" is perfectly clear to a human reading the thread and completely empty as a search query. Its embedding points at nothing in particular, so you retrieve near-noise and then blame the index.

Fix: before retrieving, rewrite the latest message into a query that stands alone, using the conversation to resolve every pronoun and every implicit reference.

import anthropic

client = anthropic.Anthropic()

history = (
    "User: which regions support same-day delivery?\n"
    "Assistant: London and Manchester."
)
question = "what about the second one?"

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=100,
    system=(
        "Rewrite the user's latest message as a standalone search query. "
        "Resolve every pronoun and implicit reference using the conversation. "
        "If the message is already standalone, return it unchanged. "
        "Return the query only, with no explanation."
    ),
    messages=[{"role": "user", "content": f"{history}\nUser: {question}"}],
)

print(message.content[0].text)
# -> "same-day delivery in Manchester"

Two details. The instruction to return an already-standalone query unchanged stops the model from mangling good queries, which is the common failure here. And max_tokens is small on purpose - a search query is a few words, and a low cap makes it hard for the model to start explaining itself.

3Choosing, and the way it goes wrong

If you build only one, build conversational rewriting - not because it is the cleverest but because without it a chat interface is broken from turn two onwards.

The costs, stated plainly

Latency. The rewrite sits in front of retrieval, and retrieval sits in front of generation. The user waits for all three. A rewrite call adds hundreds of milliseconds before the search has even started. Use a small fast model for the rewrite; you are not asking for deep reasoning.

Money. One extra call per query, on every query, forever.

Drift. This is the real one. A rewrite is a model output, so it can be wrong, and a wrong rewrite fails silently. The user asks about the 2023 policy, the rewrite drops the year, and you retrieve the 2019 version and answer confidently from it. Nothing errors. The logs look healthy.

Three habits that contain drift:

• Keep the original. Retrieve with both the original query and the rewrite, and fuse the two result lists. A bad rewrite then costs you some ranking, not the answer.
• Log both. Store the original and the rewrite side by side. When someone reports a bad answer, the first thing you want to see is what you actually searched for.
• Instruct against invention. Tell the rewriter to preserve every entity, date, number and constraint from the original, and to add nothing not implied by the conversation.

And measure it. Rewriting is exactly the kind of change that feels like an improvement and sometimes is not. Lesson 10 is how you find out.