Query rewriting

1.The gap between the question and the corpus

Chunking chunked your documents and Choosing an embedding model picked the model that embeds them. Both improve the corpus side of the match. This lesson is the other side.

A retriever compares one string to a corpus. If that string describes the need badly, no index tuning helps. User queries describe it badly in four specific ways.

All four are one problem: the string you searched with is not the string that would have found the answer. So change the string first.

Retrieval is only as good as the query it is given. If the search is failing, the query is a suspect, not just the index.

2.The four rewrites

Expansion. Add terms the answer is likely to contain: synonyms, spelled-out acronyms, the formal word for a colloquial one. PTO carryover becomes PTO paid time off annual leave carryover rollover. This helps the keyword half of hybrid search most, because BM25 matches literal terms and more of the right terms directly raises its chances. The 2023 Query2doc paper had a model write a short pseudo-passage and append it to the query, and reported gains for both BM25 and dense retrievers. On the dense side, piling on words can also pull the embedding toward a general topic, so test it rather than assume.

The cheap version needs no model: a synonym table and an acronym table for your domain. Fifty internal acronyms is fifty query classes fixed for the price of a file, and search engines ship this feature, such as the synonym sets in Elasticsearch (as of October 2026). Do that before you add a model call.

Decomposition. Split a multi-part question into one query per part, retrieve for each, then merge the results. RRF from Hybrid search works as the merge. Use it when the question holds "and", "versus", "difference between", or two nouns from different parts of your docs. Chunks were split by topic, so a chunk covering both halves of a comparison usually does not exist. One search returns one side, and the model answers half the question with full confidence.

The bill is context, not just latency. Take 3 sub-queries at 10 chunks of about 200 tokens each (invented sizes): 6,000 tokens reach the prompt, against 2,000 for one search. Either rerank down with Reranking or pay for the context.

HyDE. The name is Hypothetical Document Embeddings. Have the model write a plausible answer, then embed and search with that instead of the question. The reason is a shape mismatch. Your corpus is made of statements and your queries are questions. "Why is my build slow?" and "Build times increase when the dependency cache is invalidated on every run" concern the same thing but differ in length, grammar and vocabulary. A hypothetical answer reads like a passage, so it lands nearer real passages.

The answer need not be right. The HyDE paper (December 2022) expects the document to contain "false details" and relies on the encoder landing near real passages anyway. That is a bet, so test it on your corpus.

HyDE costs a full generation call before the search starts, so it is the most expensive of the four. It pays where questions and passages differ most, such as technical documentation and research papers. It pays less on FAQ-shaped corpora, which are already made of questions.

Conversational rewriting. This is the one most chat apps get wrong, and the one with the clearest payoff. "What about the second one?" is clear to a human reading the thread and empty as a search query. Its embedding points at nothing, so you retrieve near-noise and blame the index. Before retrieving, rewrite the latest message into a query that stands alone, using the conversation to resolve every pronoun and implicit reference. The code uses the Anthropic Messages API (as of October 2026).

import os
import anthropic

client = anthropic.Anthropic()
MODEL = os.environ["LLM_MODEL"]  # set to a current, small, fast model

history = (
    "User: which regions support same-day delivery?\n"
    "Assistant: London and Manchester."
)
question = "what about the second one?"

message = client.messages.create(
    model=MODEL,
    max_tokens=100,
    system=(
        "Rewrite the user's latest message as a standalone search query. "
        "Resolve every pronoun and implicit reference using the conversation. "
        "Keep every entity, date and number. If the message is already "
        "standalone, return it unchanged. Return the query only."
    ),
    messages=[{"role": "user", "content": f"{history}\nUser: {question}"}],
)

print(message.content[0].text)
# Typical output, not a recorded run; wording varies by model:
# -> "same-day delivery in Manchester"

Two details. The instruction to return a standalone query unchanged stops the model mangling good queries. And max_tokens is small on purpose: a search query is a few words, and a low cap makes it hard for the model to start explaining itself.

3.Choosing, and the way it goes wrong

If you build one, build conversational rewriting. It is not the cleverest, but without it a chat interface is broken from turn two onwards.

What it costs. The rewrite sits in front of retrieval, and retrieval sits in front of generation, so the user waits for all three. Measure the added latency on your own stack; a small fast model keeps it down, since a rewrite needs no deep reasoning. Money is one extra call per query, on every query, forever.

The failure case: silent drift. A rewrite is a model output, so it can be wrong, and a wrong rewrite fails silently. The user asks about the 2023 refund policy, the rewrite becomes "refund policy", and you retrieve the 2019 version and answer confidently from it. Nothing errors and the logs look healthy.

Here is how keeping the original contains it (invented ranks). The original query ranks the 2023 page first and the 2019 page fourth. The rewrite ranks the 2019 page first and the 2023 page second. With RRF at k = 60, the 2023 page scores 1/61 + 1/62, about 0.0325. The 2019 page scores 1/64 + 1/61, about 0.0320. The right page wins by a hair. That is insurance, not a guarantee, but the rewrite alone would have put the wrong page on top.

Three habits contain drift:

• Keep the original. Retrieve with both the original and the rewrite, and fuse the two lists.
• Log both. Store the original and the rewrite side by side. When someone reports a bad answer, you want to see what you actually searched for.
• Instruct against invention. Tell the rewriter to preserve every entity, date, number and constraint, and to add nothing the conversation does not imply.

When not to use it. Skip rewriting when queries already look like the documents: error codes, part numbers and exact titles, where BM25 already ranks the answer first. Skip it on a single-shot search box with strong hybrid search, and when the latency budget has no room for a model call. Try the lookup table first.

When an agent writes its own queries, as in Retrieval as a tool the agent calls, rewriting becomes the agent's job: the retriever receives whatever the agent writes. Whichever route you take, rewriting feels like an improvement and sometimes is not, so Measuring retrieval is how you find out.