Permissions that travel with the data

1The index has its own idea of who can see what

Every access-controlled corpus has two sources of truth about permissions, and they drift apart the moment nobody is watching. One lives in the system of record - the folder's sharing settings, the group membership in your identity provider, the row-level policy in the source database. The other lives in your index, as metadata you wrote onto each chunk when you ingested it. Metadata and filtering covers what happens at query time once that metadata exists: a WHERE clause or store filter that a probabilistic model can't talk its way around. None of that helps if the metadata itself is wrong.

And it goes wrong constantly, because permissions in most organizations are not a fact about a document - they're a fact about a relationship that keeps changing. A person joins a team, a folder gets reshared, a contractor's access expires, a document moves from a restricted space to a public one. The source system updates instantly, because that's its whole job. Your index updates only when something tells it to.

The gap between those two columns is where the incident happens. It's rarely a bug in the filter itself - the WHERE clause runs exactly as written. It's that the value it's checking against was true a week ago.

A permission filter is only as correct as the metadata it reads. Stale metadata behind a perfect filter is still a leak.

2Getting the bits in: model relationships, not a snapshot

The naive approach is to write a list of user IDs onto each chunk at ingestion time - "these four people can see this." It works on day one and rots from day two, because it has to be rewritten every time group membership changes anywhere near the document, and there is no trigger telling you that happened.

The fix that holds up is the one Google's Zanzibar paper popularized well beyond Google: separate the relationship from the resolution. Store the group or role a chunk belongs to - allowed_group_ids: ["team-finance", "role-manager"] - as metadata written once at ingestion. Resolve which groups the current user belongs to live, at query time, from whatever your identity provider says right now. The filter then becomes a set intersection: does the user's current group list overlap the chunk's stored group list. You've turned a fact that changes constantly (who is in which group) into a lookup you do fresh every time, and left only the far more stable fact (which groups can see this document) baked into the index.

This is exactly the model SharePoint- and Drive-backed search systems converge on in practice: permissions are materialized into the index as group or role identifiers at ingestion, and the identity provider is consulted live for the user's current memberships. Nobody tries to keep a per-user list in sync with an identity system that changes by the hour.

# ingestion time: write the relationship, not a snapshot of who currently holds it
def build_chunk_metadata(document, chunk):
    return {
        "doc_id": document.id,
        "chunk_id": chunk.id,
        "allowed_group_ids": document.acl.group_ids,   # stable-ish
        "visibility": document.acl.visibility,          # "restricted" | "org" | "public"
        "acl_synced_at": now_utc(),
    }

# query time: resolve the user fresh, then intersect
def visible_group_ids(user, identity_client):
    return set(identity_client.get_current_groups(user.id))  # never cached across requests

def is_visible(chunk_metadata, user_group_ids):
    if chunk_metadata["visibility"] == "org":
        return True
    return bool(set(chunk_metadata["allowed_group_ids"]) & user_group_ids)

Notice what never appears here: a list of individual users. Individuals move between groups constantly; groups themselves change composition far less often than group membership does, which is precisely why this split is worth making.

Nested groups make this harder than the code above lets on. A user rarely belongs to just one flat group - they inherit access through a team, which inherits through a department, which might inherit through an org-wide role. If your identity provider resolves that whole chain for you, get_current_groups can return the fully expanded set and the intersection check above still works unchanged. If it doesn't, you have to walk the chain yourself before comparing, and it's worth testing that walk explicitly rather than assuming a nested group "just works" the same way a flat one does - it's the single most common place a permission sync looks correct in a demo and leaks in production, because the demo never had more than one level of nesting to get wrong.

3Keeping it true: sync permissions on their own clock

Content freshness and permission freshness are not the same problem and should not share a sync schedule. A blog post being a day stale is an inconvenience. A revoked user still retrieving a document is a security incident. Most source systems - Google Drive, SharePoint, an internal admin tool - expose a changes feed or a webhook specifically for this, separate from the feed that tells you a document's text changed. Use it.

Poll where you must, but treat revocation as the case that decides your sync interval, not the average case. A stale group-membership cache saves a few API calls and costs you exactly the incident this lesson opened on. And when a sync run fails - the identity API times out, a webhook payload doesn't parse - fail loudly. A permission sync that silently skips a batch looks identical, from the outside, to one that succeeded. Log the failure, alert on it, and treat "we don't know if this ACL is current" as equivalent to "this ACL is wrong" until proven otherwise.

The only way to know any of this actually works is to test the failure case directly: revoke a real test user's access in the source system, wait exactly as long as your stated freshness target allows, then search as that user. If the document still comes back, you don't have a sync job, you have a sync job that has never been checked.

Pick the freshness target as deliberately as you would for content changes - by asking what staleness actually costs, not by inheriting whatever a cron job happens to run - and write it down somewhere a security review can find it - "revoked access is reflected in search within five minutes" is a claim you can audit, and "we sync permissions periodically" is not. The two targets don't have to use the same mechanism. It's entirely normal to poll content changes nightly while consuming permission-change events in near real time, because the two failure modes cost completely different amounts. A pipeline that treats them identically is usually treating permissions too slowly, not content too fast.