From source system to corpus

This lesson is about the step before a file exists on your disk at all - getting it out of a wiki, a helpdesk, a CRM, or a shared drive that was never built to be read by a pipeline. Parsing documents before you chunk them, next, is about what happens once that file is in hand. Skip this step, and no amount of layout-aware parsing helps, because the document never arrives.

1Getting bytes out is a different job than parsing them

A PDF sitting in a folder is the easy case. Something already put it there, and your job is only to read it correctly. Most of a real corpus does not arrive that way. It lives inside a running system - a wiki with a permissions model, a ticketing tool with a rate-limited API, a CRM that only shows you records through its UI or a paid export - and getting content out means talking to that system on its terms, not yours.

Each kind of source system fails in its own way, and the failure has nothing to do with file format.

None of these problems show up in a parsing library, because parsing runs after extraction. If your retrieval quality looks bad and the corpus is missing whole documents rather than mangling the ones it has, look here first, not at chunk size.

2Buy the connector, hand-roll only the gap

Once you accept extraction as its own problem, the next question is who solves it: a connector someone else maintains, or a script you write and own.

A connector is a piece of software built specifically to pull data out of one kind of system, maintained by someone who watches that system's API for changes. Airbyte and Fivetran are the general-purpose examples; Unstructured and similar libraries ship source connectors for wikis, drives and ticketing tools aimed specifically at feeding an LLM pipeline. Using one means someone else has already solved pagination, retry-on-429, and incremental fetch for that exact system, and keeps solving it as the system's API drifts.

A hand-rolled script is code you write against the source's API directly. It is faster to start - no new dependency, no configuration format to learn - and it can do exactly the narrow thing you need and nothing else. It is also entirely your job to keep working when the source system changes anything.

The right default is to reach for a connector first and hand-roll only the gap it does not cover - a niche internal tool, a one-off migration, an endpoint too narrow for a general connector to be worth adopting. Where a script is genuinely the right call, write it as if someone else has to run it without you: log every failure, never let a partial fetch look like a successful one, and never hard-code something about the source's structure - a folder path, a fixed set of IDs - that the source is free to change without telling you.

import hashlib
import time

import requests

def extract_pages(base_url: str, api_token: str, since_cursor: str | None = None):
    """Pull every page from a wiki-style API, one batch at a time.

    Real extraction code, whether hand-rolled or inside a connector,
    has to handle three things a parser never sees: pagination, rate
    limits, and detecting which pages actually changed.
    """
    headers = {"Authorization": f"Bearer {api_token}"}
    cursor = since_cursor
    documents = []

    while True:
        resp = requests.get(
            f"{base_url}/pages",
            headers=headers,
            params={"cursor": cursor, "limit": 100},
            timeout=30,
        )
        if resp.status_code == 429:
            wait = int(resp.headers.get("Retry-After", 5))
            time.sleep(wait)
            continue
        resp.raise_for_status()

        batch = resp.json()
        for page in batch["results"]:
            content_hash = hashlib.sha256(page["body"].encode()).hexdigest()
            documents.append({
                "source_id": page["id"],
                "title": page["title"],
                "body": page["body"],
                "content_hash": content_hash,
                "source_url": page["url"],
            })

        cursor = batch.get("next_cursor")
        if not cursor:
            break

    return documents

Notice what the function returns: not just text, but a content_hash and a source_url alongside it. That is not incidental - it is the beginning of a schema every document needs before it is ever chunked: a stable id, a content hash, and a source url decided at the point of extraction, and it is cheapest to capture right here rather than reconstructed later from a corpus that has already forgotten where anything came from.

3Where the ingestion budget actually goes

Teams that budget an ingestion project usually put the biggest number next to embedding, because that is the line item with a per-token price attached to it. In practice the money and the time go somewhere else.

Embedding a chunk costs the same whether that chunk came from a PDF, a wiki page, or a support ticket - it is one uniform API call regardless of source. Extraction has none of that uniformity. Every source system has its own auth model, its own pagination scheme, its own way of telling you a record changed, and its own way of silently going quiet when something upstream shifts. That variability, not the per-token price of an embedding call, is what actually consumes an ingestion team's time.

The embedding call looks the same for every document. Getting the document is different every single time - that is where the budget goes.

This is worth saying plainly before you scope a project, because it changes what you plan for. A four-source corpus is not one extraction problem times four - it is four different extraction problems, each with its own way of breaking, and each needing someone to notice when it does.