Metadata and filtering

Similarity gets all the attention, and it is maybe half of what a retrieval query does. The other half is structured and boring: this tenant, this language, this document type, published after this date, visible to this user. None of that is fuzzy. All of it has to be exactly right, and one item on that list is a security boundary.

1Where the filter runs decides whether it works

Three places a filter can happen, and they are not interchangeable.

Post-filtering is the trap, and it is easy to fall into because it is the natural way to write the code. Ask for 10, get 10 back, filter — and now you have 2. Nothing errored. The user sees a thin answer. If your corpus has 50 tenants and you post-filter by tenant, most of your top 10 belongs to other tenants and your effective k is closer to 1.

The fix is to express the filter as part of the query so the store handles it. In SQL that is simply a WHERE clause next to the ORDER BY on distance. In a dedicated store it is the filter argument in the search call. Either way the store, not your application, decides how to combine the constraint with the index.

That combination is genuinely hard, which is worth knowing so the behaviour does not surprise you. An HNSW graph was built over every vector, so a narrow filter means the walk keeps arriving at nodes it must reject. Stores handle this differently: some maintain filterable index structures, some detect a selective filter and fall back to an exact scan over the matching subset — which is the right answer, because a few thousand rows do not need an index. In Postgres, a WHERE tenant_id = $1 that matches a small fraction of rows will often be served better by a plain index scan plus exact distance than by HNSW, and the planner may or may not choose well. Check with EXPLAIN ANALYZE on real data rather than assuming.

-- Pre-filter: constraints and similarity in one query, top 10 guaranteed.
SELECT id, doc_id, body,
       1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE tenant_id  = $2                             -- hard boundary
  AND $3 = ANY(allowed_group_ids)                 -- hard boundary
  AND lang       = $4
  AND doc_type   = ANY($5)
  AND published_at <= now()
ORDER BY embedding <=> $1
LIMIT 10;

-- Recency as a preference, not a cutoff: blend similarity with age decay.
SELECT id, body,
       (1 - (embedding <=> $1))
         - 0.15 * (1 - exp(-EXTRACT(epoch FROM now() - updated_at) / (86400 * 180)))
       AS score
FROM chunks
WHERE tenant_id = $2 AND $3 = ANY(allowed_group_ids)
ORDER BY score DESC
LIMIT 10;

The second query is a starting point, not a formula to copy blindly. The half-life and the 0.15 weight are yours to set, and the only way to set them is to try a couple of values against real queries.

2Permissions are not a preference

Everything above is about quality. This part is about not having an incident.

If a chunk is retrieved, it is exposed. It goes into the context window, the model reads it, and the answer can quote it, paraphrase it or leak it — and even if the answer is clean, the retrieved chunks are usually logged, traced, and shown in a citations panel. There is no step after retrieval that makes an unauthorised chunk safe.

So: the permission filter is part of the retrieval query, enforced by the store. Not a line in the system prompt. Not a check on the generated answer. Not a hope that the model behaves. A prompt instruction is a suggestion to a probabilistic system; a WHERE clause is a constraint.

Two practical notes. First, permissions change and vectors do not: if access is derived from groups that change often, store the group ids on the chunk and resolve the user's groups at query time, rather than baking a user list into the row. Second, decide deliberately what an empty result means. A user who is allowed to see nothing should get "I could not find anything", and your logs should record that the filter emptied the candidate set — otherwise you cannot tell a permissions bug from a retrieval bug.

A filter that protects data is a database constraint. If it lives in a prompt, it is not a control, it is a wish.

3What to store, and store it now

Metadata you did not capture at index time is usually gone. The source document has moved, been edited, or been replaced, and no amount of reprocessing will tell you what it said when you ingested it. Since a metadata column costs nothing next to a 1024-dimension vector, err heavily on the side of storing more.

That last row is the one people leave out and regret, because without it a half-finished re-embed leaves you with a table you cannot reason about.

Two habits that follow from having good metadata. Filter fields should be indexed like any other database column — a tenant_id with no B-tree index will make your filtered search slow in a way that looks like the vector index is at fault. And keep an eye on cardinality: filters that match a handful of rows are best answered by an exact scan over those rows, while filters that match nearly everything may as well not be there. Knowing roughly where each of your filters sits on that spectrum tells you what to expect from the query planner, which is the difference between tuning and guessing.