Iterative retrieval

1.When one pass, even a decomposed one, is not enough

Query rewriting already covers a question with several parts. Split "how does our refund policy differ from our exchange policy" into two searches, run both, merge the results. That works because both queries can be written before either one runs.

Some questions do not allow that. "Find the incident that caused this regression, then tell me who reviewed the fix" hides a second search, but its query depends on an answer you do not have yet. You cannot write "review of incident 4471" until the first search says the incident was 4471. "Which supplier missed the deadline, and what penalty did we agree" has the same shape. Each is a dependent lookup: the second query is a function of the first result.

The idea predates today's agent tooling. A December 2022 paper, IRCoT (Trivedi et al., ACL 2023), alternates one reasoning sentence with one search, using the newest sentence as the next query. It used BM25 over Wikipedia-based multi-hop question sets, and it still capped itself: the loop ended on an answer or after 8 steps. Even a research loop needed a ceiling. When RAG is the wrong tool names this shape of question in passing; this lesson works it through.

Decomposition answers a question you can fully specify in advance. Iterative retrieval answers one you cannot: you learn what to search for next only by reading what came back.

2.The loop: search, read, decide

The loop adds no new way to search. Every tool inside it is one you already have: hybrid search from Hybrid search, a reranker from Reranking to narrow the candidates, and rewriting from Query rewriting to turn the next need into a real query. What is new is the round trip around them.

Each round does three things:

1. Search. Run the current query through your normal retrieval stack. Nothing about this step changes because it sits inside a loop.
2. Read. Give the new chunks to the model, together with what earlier rounds gathered.
3. Decide. It is not the general agent choice of what to do next. It is narrower: whether the evidence so far is enough to answer and cite, or whether another search is needed. That is coverage, not task management. Keep it narrow and the loop stays small; let it drift into a general choice of actions and you have built the model-driven loop of Retrieval as a tool the agent calls for a job that did not need it.

# pip install anthropic
import os
import anthropic

client = anthropic.Anthropic()
MODEL = os.environ["LLM_MODEL"]  # set to a current model
MAX_ROUNDS = 4
TOKEN_BUDGET = 20_000  # input + output tokens, summed over every decide call

def retrieve(query, user):
    # Your hybrid search + rerank; returns objects with .id and .text.
    # The user's group filter runs on every call, so every round is checked.
    ...

def decide(question, gathered_chunks):
    resp = client.messages.create(
        model=MODEL,
        max_tokens=200,
        system=(
            "Decide whether the retrieved evidence is enough to answer the "
            "question with a citation. If yes, reply STOP followed by the answer. "
            "If not, reply SEARCH followed by the next query to run."
        ),
        messages=[{
            "role": "user",
            "content": f"Question: {question}\n\nEvidence so far:\n{gathered_chunks}",
        }],
    )
    return resp

def iterative_retrieve(question, user):
    query = question
    gathered = []
    seen_chunk_ids = set()
    spent = 0

    for round_n in range(MAX_ROUNDS):
        chunks = retrieve(query, user)
        new_ids = {c.id for c in chunks} - seen_chunk_ids

        if not new_ids and round_n > 0:
            break  # thrash: nothing new came back, so stop

        seen_chunk_ids |= new_ids
        gathered.extend(c.text for c in chunks if c.id in new_ids)

        resp = decide(question, "\n---\n".join(gathered))
        verdict = resp.content[0].text
        if verdict.startswith("STOP"):
            return verdict.removeprefix("STOP").strip()
        spent += resp.usage.input_tokens + resp.usage.output_tokens
        if spent >= TOKEN_BUDGET:
            break  # fail loudly below, do not truncate silently
        query = verdict.removeprefix("SEARCH").strip()

    return "Could not find enough evidence within the round or token limit."

decide has no tool menu: it reads evidence and either stops with an answer or names the next search. The code keeps every chunk; dropping the irrelevant ones is the hygiene step below. Two more details matter.

Permissions run every round. Each round is a new query, so it needs the same group filter as the first one; Permissions that travel with the data covers how. If you check the source live, cache a verdict per user and document for one request (a design suggestion, not a measured result).

Context hygiene. Every round adds text, and text the loop has already judged irrelevant still costs tokens on every later read. Chroma, a vector database vendor, reported in July 2025 that even a single distractor passage lowered accuracy across 18 models of that date (2025) on long-context needle tests (vendor-reported, mechanism unknown; the authors do not explain it). So have decide also name chunks it found irrelevant, and delete them from gathered. Keep their ids in seen_chunk_ids so they cannot return.

3.What bounds the loop, and what it costs

A loop with no stopping rule does not fail loudly; it keeps spending. Three limits bound it, each catching a different failure.

• A hard round cap. MAX_ROUNDS above. However the decision reasons, it does not get unlimited turns. Your code enforces it; the model does not choose to respect it. The Agents course's Planning a run, and knowing when to stop covers the wider budget.
• A token budget. Each round is a search plus a model call, and the call re-reads everything gathered. Track tokens per request and fail loudly at the limit.
• Thrash detection. The same chunk comes back with nothing new. That is not a sign you are one search from the answer; the corpus has given up what it has on this query. The new_ids check in the code is exactly this.

What it costs. One pass sends 1,000 tokens. The two-round run above has decide read 1,000 + 2,000 = 3,000 tokens, three times as much. Dropping irrelevant chunks makes that 1,000 + 1,200 = 2,200. The gap widens quickly, because round n re-reads rounds 1 to n. Four rounds with nothing dropped read 1,000 + 2,000 + 3,000 + 4,000 = 10,000 tokens, ten times one pass. If only one 200-token chunk survives each round, the same four rounds read 1,000 + 1,200 + 1,400 + 1,600 = 5,200.

Two named failures. The first is thrash, covered above. The second is the early stop on a near miss. Round 1 returns a chunk about incident 4412, a similar regression from last month. It reads like an answer, decide says STOP, and the reply cites the wrong incident. Nothing errored. The fix is to make decide quote the sentence that supports each part of the question before it may stop, so a missing reviewer is visible. Loops with no cap are easy to write by accident: the official LangGraph tutorial for a grade-and-rewrite retrieval agent (as of October 2026) adds no round counter of its own in its code. LangGraph's default limit (1,000 steps, as of October 2026) is far too high to be a budget and ends in an error, so set your own.

When not to use it. Skip the loop when decomposition can name every query up front: it costs one extra call, not one per round. Skip it when one well-rewritten query already covers the ground. Skip it on a latency-bound path, where each round is a search plus a model call the user waits for. Anthropic's December 2024 essay on building agents advises adding complexity "only when it demonstrably improves outcomes". Test the loop against your single-pass baseline on questions that need hops, using the method in Measuring retrieval.

A loop follows one path. A question about the whole corpus, such as the common themes across a year of incidents, has no path to follow, and When RAG is the wrong tool covers what to do instead. When you want the model, not your code, to choose the tool and the number of searches, that is Retrieval as a tool the agent calls.