Citations and grounding

1An id is a handle

Lesson 05 showed the shape: put an id next to each passage, ask the model to cite it. This lesson is about making that hold up in production.

Start with the ids themselves, because a bad id makes everything downstream useless.

The test is simple: six months from now, someone pastes a cited id into a search box. Does it find the exact passage? If not, the citation is decoration.

Store the id with the chunk at index time, alongside the metadata from lesson 06, and carry it through retrieval and reranking untouched. It should be the same string in your index, your prompt, your logs and your UI.

The format in the prompt only needs to be unambiguous and easy to parse:

[refund-policy-digital] Refunds on digital goods are available within 14 days of purchase.

[refund-policy-physical] Physical goods may be returned within 30 days if unopened.

Square brackets, one blank line between chunks. Nothing clever. Whatever you choose, keep it consistent, because your verifier is going to parse it.

A citation is not a promise from the model. It is a handle you can pull on in code.

2The prompt, including permission to fail

Three instructions do most of the work, and the third is the one people leave out.

Answer only from the passages below. Sets the boundary.

After each claim, cite the id it came from, and quote the words you used. The quote is what makes verification possible. Without it you can check that the id exists; with it you can check that the id supports the claim.

If the passages do not answer the question, say exactly that and stop. This is the important one. A model with no permitted way to fail will not fail - it will produce something. Naming the failure output, and giving it exact words to use, converts confident invention into a signal you can count and alert on.

Be specific about the failure case rather than gesturing at it:

That last row matters more than it looks. Cross-passage synthesis is where citations get slippery - the model takes half a fact from one chunk and half from another and cites both, and each citation is individually valid while the combined claim is not in either passage.

Some APIs now handle part of this for you. Anthropic's citations feature has the model return structured references to the document spans it used, rather than you parsing brackets out of prose. If you are on a platform that offers it, prefer it - structured output is easier to verify than text you have to regex. The reasoning in this lesson does not change either way.

3Verifying, and what it cannot tell you

Here is the check. It parses each quoted span with the id cited next to it, and confirms the words are really in that chunk.

import re

chunks = {
    "policy-14": "Refunds on digital goods are available within 14 days of purchase.",
    "policy-15": "Physical goods may be returned within 30 days if unopened.",
}

answer = 'No. Digital goods are "available within 14 days of purchase" [policy-14].'

def normalise(text):
    return re.sub(r"\s+", " ", text).strip().lower()

# every quoted span must appear in the chunk cited right after it
for quote, chunk_id in re.findall(r'"([^"]+)"\s*\[([^\]]+)\]', answer):
    chunk = chunks.get(chunk_id)
    if chunk is None:
        print(f"FAIL  cited {chunk_id}, which was not in the prompt")
    elif normalise(quote) not in normalise(chunk):
        print(f"FAIL  {chunk_id} does not contain: {quote}")
    else:
        print(f"OK    {chunk_id}")

# -> OK    policy-14

Normalising whitespace and case before comparing matters, because models reflow text. If your corpus has heavy formatting you may want to strip punctuation too, or fall back to a fuzzy ratio - but keep the threshold high, or you are back to guessing.

What to do when it fails is a product decision, not a technical one:

Run this on every response in production and count the failures. That rate is a real signal: it moves when you change the prompt, the model or the chunk size, and a jump in it is an alert worth having.

Citations are also a feature

Do not treat this as purely internal machinery. Rendering the cited passage next to the answer - as a link, a hover, a side panel - changes what your product is. A confident paragraph asks the reader to trust you. A confident paragraph with the source one click away lets them check you, which is a different and much better offer.

It also changes behaviour in a way worth noticing. Users who can see the sources spot bad answers themselves, and they report them with the passage attached. Your bug reports arrive pre-diagnosed.

The honest limit

Verification proves a narrow thing: the quoted words exist in the chunk that was cited. It does not prove the answer is right.

All of these pass the check:

• The passage says refunds are available within fourteen days for members. The answer drops "for members" and cites the passage. Every quoted word is present.
• Two passages each supply half a fact. Both citations verify. The combined claim appears in neither.
• The passage is a heading or a table of contents entry that names the topic without stating anything. The quote is real; the support is not.
• The passage is out of date. It says what the model claims. It has simply been superseded, which is a metadata problem from lesson 06, not a citation one.

So what have you actually bought? You have eliminated the crudest and most common failure - text with no source at all - and you have made every remaining answer traceable to a passage a human can read in seconds. That is a large gain and it is not the same as correctness. Correctness still needs the evaluation from lesson 10, and for anything consequential it still needs a person.