RAG without the hype

1Three steps, and nothing else

RAG stands for retrieval-augmented generation. Strip the name away and it is three steps:

1. Retrieve. Take the user's question, search your documents, get back the handful of passages most likely to contain the answer. This is the search you built in lesson 04 — embeddings, cosine similarity, chunking, hybrid search.
2. Stuff. Paste those passages into the prompt as plain text, above the question.
3. Answer. Send it to the model and ask it to answer using only what you pasted.

There is no fourth step. No training, no model changes, no special API. The model never learns your documents; it reads them fresh in the prompt each time, the same way it reads anything else you send.

That framing matters because it tells you where the failure can be. Only two places: the passages you retrieved, or the way you asked. Nothing else in the pipeline can be wrong, because nothing else exists.

The model does not know your documents. It reads them. Everything you retrieve is everything it can possibly know.

Why not fine-tune

Fine-tuning means continuing the model's training on your own examples so the new behaviour is baked into the weights. People reach for it when they hear "the model should know our docs". It is almost always the wrong choice for that goal.

Fine-tuning is for teaching the model how to behave. RAG is for telling it what is true right now. Most of the time you want the second one.

2Your answer cannot beat your retrieval

This is the single most useful idea in the lesson, so it gets its own section.

If the correct passage was not in the top results, no prompt fixes it. Not a better system prompt, not a bigger model, not "think step by step". The information simply is not in the context window. The model will either say it does not know, or it will fill the gap from its training data — which is exactly the invented answer you were trying to avoid.

So when a RAG answer is wrong, do this before touching anything else:

• Print the chunks that retrieval returned for that exact question.
• Read them yourself. Is the answer in there?
• If yes, it is a prompt problem. Go fix the instructions.
• If no, it is a retrieval problem. Do not touch the prompt.

Most teams get this backwards and spend a week tuning prompt wording against passages that never contained the answer.

Measure retrieval on its own. Take fifty real questions, note which chunk actually answers each one, and check how often that chunk lands in the top k results. That number is recall@k — the share of questions whose correct passage appears in the top k retrieved chunks. Recall@5 of 0.6 means forty percent of your questions are unanswerable before the model is even called.

The cheap large win: reranking

Vector search is fast and rough. It compares one embedding per chunk, so it gets you into the right neighbourhood but often puts the best passage at position seven.

A reranker fixes the ordering. It is a smaller, slower model that looks at the question and one chunk together and scores how well that chunk answers it. Because it reads both at once it is far more accurate than comparing two embeddings — and because you only run it on candidates search already found, it stays cheap.

The pattern:

• Retrieve 50 candidates with vector or hybrid search. Fast, generous, imprecise.
• Rerank those 50, keep the top 5. Slow, narrow, accurate.
• Send those 5 to the model.

Retrieve wide, rerank narrow. For most apps this is the biggest quality jump you can get for an afternoon of work.

3Grounding, citations, and when to walk away

You want two things from the answer: it should come from the passages, and you should be able to prove it did.

Give every chunk an id and put the id in the text you paste. Then require the model to cite it. Now a wrong answer is visible: if it cites [doc-14] and doc-14 says nothing of the kind, you caught it with an automatic check, not a human reading every response.

The three prompt instructions that do most of the work:

• Answer using only the passages below.
• After each claim, cite the chunk id it came from.
• If the passages do not contain the answer, say so. Do not guess.

That last one is the one people forget, and it is the one that turns confident nonsense into a useful "I do not know".

import anthropic

client = anthropic.Anthropic()

# chunks came from your search step - each one carries an id you can check later
chunks = [
    {"id": "policy-14", "text": "Refunds on digital goods are available within 14 days of purchase."},
    {"id": "policy-15", "text": "Physical goods may be returned within 30 days if unopened."},
]

passages = "\n\n".join(f"[{c['id']}] {c['text']}" for c in chunks)

message = client.messages.create(
    model="claude-sonnet-5",
    max_tokens=500,
    system=(
        "Answer only from the passages provided. "
        "Cite the chunk id in square brackets after each claim. "
        "If the passages do not contain the answer, say you do not know."
    ),
    messages=[{
        "role": "user",
        "content": f"Passages:\n{passages}\n\nQuestion: Can I get a refund on an ebook after 20 days?",
    }],
)

print(message.content[0].text)
# -> "No. Refunds on digital goods are limited to 14 days. [policy-14]"

Two details worth noticing. The rules live in the system prompt and the data lives in the user turn — that separation from lesson 02 keeps a document from quietly issuing instructions. And max_tokens is small, because a grounded answer over five passages should be short.

When RAG is the wrong tool

RAG retrieves passages. If the answer is not sitting in some passage, retrieval has nothing to hand over.

The pattern: anything that needs aggregation over everything is a database question, not a retrieval question. Run SQL, or a metadata filter, or read the documents in full in a loop. Reaching for RAG there gives you an answer built from five arbitrary passages, stated with total confidence, and wrong.

Know which kind of question you have before you build. That choice matters more than any tuning you do afterwards.