From source system to corpus

A support lead asks why the assistant knows nothing about last quarter's outages. The postmortems are in the wiki. The parser is fine and the chunker is fine, but the wiki was never indexed: someone ran an export in March and nobody ran it again. This lesson is about the step before a file exists - getting content out of a system that was never built to be read by a pipeline. Parsing documents before you chunk them, next, takes the bytes from there.

1.Getting bytes out is a different job than parsing them

A PDF in a folder is the easy case. Something already put it there, and your job is only to read it correctly. Most of a real corpus does not arrive that way. It lives inside a running system with its own permissions, its own limits and its own idea of what a page is, and you get content out on that system's terms. Each kind fails in its own way, and none of it has to do with file format. The systems named here are examples, current as of October 2026.

None of this shows up in a parsing library, because parsing runs after extraction. If the corpus is missing whole documents instead of mangling the ones it has, look here first, not at chunk size.

Rate limits are the part that surprises people, so do the arithmetic before you design anything. Slack's changelog says that since May 2025, new commercially distributed apps outside its Marketplace, and new installations of them, may read channel history at one request per minute, with at most 15 messages per request (as of October 2026). Whether older installs are covered is unclear: Slack's changelog says they are not, but other sources report a move to the same limits in March 2026, so check Slack's current page for your install. Internal apps you build for your own workspace keep much higher limits. Say you ship a distributed app, and one customer's workspace holds 100,000 messages. That count is invented, but the division is not.

The third row is the failure to name. Call it the quiet partial crawl: nothing errors, the corpus is simply short, and retrieval blames the model for pages that were never there. It gets worse downstream, because a pipeline that treats a missing page as a deleted page will turn a short crawl into a mass deletion. The guard is a count: compare what you fetched with the total the source reports, or with the last run, and fail when it falls short.

2.Buy the connector, hand-roll only the gap

A connector is software built to pull data out of one kind of system, maintained by someone who watches that system's API change. Airbyte is a general example, an open-source platform whose docs list over 600 sources and destinations (as of October 2026), mostly databases and SaaS tools. Unstructured, built for preparing documents for LLM pipelines, lists Confluence, SharePoint and Slack among its source connectors (as of October 2026). Either way, pagination, retry-on-429 and incremental fetch are already solved for those systems.

A hand-rolled script is code you write against the source's API. It starts faster, does exactly the narrow thing you need, and is entirely yours to keep working.

Default to a connector and hand-roll only the gap: a niche internal tool, a one-off migration, an endpoint too narrow to justify adopting one. Where a script is the right call, write it for someone who has to run it without you. Log every failure, never hard-code something the source may change (a folder path, a fixed set of IDs), and fail on a short crawl.

import hashlib, json, logging, time
import requests

log = logging.getLogger("extract")

def sha(text: str) -> str:
    return hashlib.sha256(text.encode()).hexdigest()

def extract_pages(base_url: str, api_token: str, cursor: str | None = None):
    """Pull every page from an invented wiki-style API (invented response shape)."""
    headers = {"Authorization": f"Bearer {api_token}"}
    documents, retries = [], 0

    while True:
        resp = requests.get(f"{base_url}/pages", headers=headers, timeout=30,
                            params={"cursor": cursor, "limit": 100})
        if resp.status_code == 429:
            retries += 1
            if retries > 5:
                raise RuntimeError("still rate limited after 5 retries")
            # Retry-After is seconds or an HTTP date; this reads only seconds
            wait = resp.headers.get("Retry-After", "")
            log.warning("429 at cursor %s, retry %d", cursor, retries)
            time.sleep(int(wait) if wait.isdigit() else 30)
            continue
        resp.raise_for_status()
        retries = 0

        batch = resp.json()
        for page in batch["results"]:
            meta = {"title": page["title"], "acl": page["groups"]}
            documents.append({
                "source_id": page["id"],
                "source_url": page["url"],
                "body": page["body"],  # the source text, kept whole
                "text_hash": sha(page["body"]),
                "meta_hash": sha(json.dumps(meta, sort_keys=True)),
                "meta": meta,
            })
        cursor = batch.get("next_cursor")
        if not cursor:
            break

    if len(documents) < batch["total"]:  # the quiet partial crawl
        raise RuntimeError(f"fetched {len(documents)} of {batch['total']}")
    return documents

Notice what comes back besides text. A stable id and a source url are decided here, because a corpus that has forgotten where things came from cannot be repaired later. The url also lets you ask the source who may read a page, a check Permissions that travel with the data builds on. The body is kept whole, not only the chunks cut from it: a new chunker, a model that reads whole documents (Giving each chunk its context) or a new embedding model (Changing the embedding model underneath a live index) all need the original text, and by then the page may have changed or gone.

The two hashes are the other half. Hash the text and the metadata separately: the text hash decides whether a vector must be recomputed, the metadata hash whether stored fields must be patched. After chunking, keep the same pair per chunk. When a page disappears from a later crawl, record a tombstone, a marker that it is gone, instead of silently forgetting it - but only after the count check has passed, so a short crawl never turns into deletions. Keeping the index fresh without paying twice shows how the pair and the tombstone drive updates and deletes.

3.Where the ingestion budget actually goes

Many teams budget an ingestion project by putting the biggest number next to embedding, because that line has a per-token price. The first section showed where the time goes instead.

Embedding a chunk costs the same whether it came from a PDF, a wiki page or a ticket: one uniform call. Extraction has none of that uniformity. Every source has its own auth model, its own pagination, its own way of saying a record changed, and its own way of going quiet when something upstream shifts. That variability, not the per-token price, is what consumes an ingestion team.

The embedding call looks the same for every document. Getting the document is different every single time - that is where the budget goes.

When not to build this. If the content changes rarely and the system offers a one-time export, export once and treat the dump as your source. A sync pipeline costs more to run than it saves, and a stale snapshot you control beats a live crawl you cannot trust. The same goes for a folder of files you already hold: read them with a parser and skip the connector. Build the pipeline when the source keeps changing and the answers must keep up.