Keeping the index fresh without paying twice

1.Five kinds of change

Every edit in a source system is one of five events, and only one of them needs a new vector. Below, a span is one chunk of a page, as Chunking defines it.

A naive pipeline treats all five as the first row. It sees "page changed", re-chunks the page and embeds every chunk again. A one-character typo costs the same as a rewrite, and a permission flip, which moves no vector, costs the same as both. Notion (as of its February 2026 write-up) described fixing this in July 2025 and reported about 70% less data volume. That is one company's own number, with no dollar figure, from a corpus far bigger than yours. Take the pattern, not the number.

One caution on "metadata moves no vector". If you stamp a title or header line into the text before embedding, as Chunking describes, that field is text for this purpose. Put it in the text hash.

An edit is not a re-embed. Ask what changed, then pay only for that.

2.The page-state record

Split each page into spans with the chunker from Chunking. For each span keep two hashes: one of the normalised text, one of every metadata field that goes into the index. Store the list, with a model and chunker version, as one record per page (the page-state record) in an ordinary key-value table. Notion keeps its records the same way, one per page.

On every event, re-chunk the page, build a new record and compare it with the stored one. Chunking is cheap computing. The embedding call is the part you are avoiding.

Here is an invented six-span page, "Refund policy". The figure walks four events through it.

The comparison is a few lines of Python. It returns a plan instead of acting, so you can test it without an embedding model.

import hashlib, json, os

# Bump the chunker part when you change the chunker. A new model id
# changes the string too, and every stored hash then counts as stale.
VERSION = os.environ["EMBED_MODEL"] + "|chunker-1"

def h(value) -> str:
    raw = json.dumps(value, sort_keys=True).encode()
    return hashlib.sha256(raw).hexdigest()[:16]

def page_state(spans):
    """spans: [{"text": ..., "meta": {...}}], in a fixed order."""
    return {"version": VERSION,
            "spans": [(h(" ".join(s["text"].split())), h(s["meta"]))
                      for s in spans]}

def plan_updates(prev, new):
    plan = {"embed": [], "patch": [], "skip": [], "delete": []}
    stale = prev is None or prev["version"] != new["version"]
    old = [] if stale else prev["spans"]
    for i, (text_h, meta_h) in enumerate(new["spans"]):
        if i >= len(old) or old[i][0] != text_h:
            plan["embed"].append(i)
        elif old[i][1] != meta_h:
            plan["patch"].append(i)
        else:
            plan["skip"].append(i)
    old_n = len(prev["spans"]) if prev else 0
    plan["delete"] = list(range(len(new["spans"]), old_n))
    return plan

spans = [{"text": f"Paragraph {i}.", "meta": {"acl": "private"}} for i in range(6)]
before = page_state(spans)
spans[3]["text"] += " Sale items are excluded."
print(plan_updates(before, page_state(spans)))
# {'embed': [3], 'patch': [], 'skip': [0, 1, 2, 4, 5], 'delete': []}

Run the plan, then save the new record last. If the job dies halfway, the old record is still there and the next run plans the same work again.

What it costs. Invented numbers: a day brings 2,000 events on six-span pages. Say 1,000 are one-span edits, 600 are permission flips and 400 change nothing you index. The naive pipeline makes 2,000 x 6 = 12,000 embedding calls. This one makes 1,000, and 600 patches. You pay for it in a state table, a re-chunk on every event, and a hash per span. All three are small beside an embedding call.

3.Deletes, drift and when to skip it

Deletes are the case an edit-driven pipeline never sees. The failure looks like this: the team removes a wrong policy page, the index keeps its vectors, and the assistant keeps quoting it with a citation that now leads nowhere. Nothing errors. Do not assume your source announces deletes. Check, and add a reconcile pass either way: list the ids in the source, list the ids in the index, and remove the difference.

Run deletes last, and guard them. Qdrant's incremental-update tutorial (as of July 2026) refuses to delete at all when the source list is empty, with the message "Refusing to delete from an empty source snapshot." A connector that fails and returns nothing must not look like a company deleting everything. A comment in its code suggests capping how many deletes one run may make. A reconcile also reads every stored hash each run, so cost grows with the corpus: the tutorial says about 100,000 points is fine and that around a million can hit its 32 MB request cap.

For ordering, a delete should leave a marker, usually called a tombstone, so a late edit event cannot bring the page back. That is general practice, not something the docs above state. turbopuffer's write API (as of October 2026) offers conditions on upserts, patches and deletes, checked against the stored document. That lets you refuse a write that is older than what the index holds, our reading and not the docs' example.

What breaks the hashes. Three things, and none of them errors:

• A chunker whose output order changes between runs. Spans are compared by position, so one edit re-embeds most of the page.
• A position shift. Insert a paragraph at the top and every later span moves down one. Qdrant's answer: when an id is new but the text hash already exists, copy the stored vector instead of embedding.
• A change of model or chunker, which makes every stored vector wrong for the new setup. The version string in the code forces the full re-embed described in Changing the embedding model underneath a live index. That is also why you keep the source text, as From source system to corpus says: a re-embed needs it.

Context lines break the per-chunk rule. If each chunk carries a generated context line, or you use a contextualized embedder, as Giving each chunk its context describes, one text edit can change what every line in that document should say. Hashing per chunk then under-counts. Plan at document level instead: when any text hash changes, regenerate and re-embed all chunks of that document. The keyword index from Hybrid search must follow the same plan. Apply each plan_updates result (embed, patch, delete) to both indexes, or the two halves drift apart.

Two paths, one planner. Notion runs single edits through a queue with sub-minute freshness (self-reported) and backfills through batch jobs. Both can call the same plan_updates. Permission patches belong on their own faster clock, as Permissions that travel with the data argues. Measure the time from an edit to the edit being searchable. Pinecone's docs (as of October 2026) say reads right after a write may not show it, so test with a delay.

When not to use it. If the corpus is small and changes rarely, rebuild it every night. A full re-embed that takes ten minutes, run once a day, is cheaper than the state table, the reconcile and the bugs in both. If staleness matters more than query cost, fetch the page when asked instead, as Retrieval as a tool the agent calls describes, and pay on every query. The state record also knows nothing about quality: an index that is perfectly fresh can still retrieve badly, and Measuring retrieval is where you find out.