Chunking

You do not retrieve documents. You retrieve chunks. Whatever your chunker produces is the complete set of things your system can ever return, so a fact that got cut in half is a fact your system cannot answer about, no matter how good the model downstream is.

Everything after this step operates on the units it creates.

1.Two forces, pulling opposite ways

A chunk has to satisfy two requirements that fight each other.

Small enough to be about one thing. A chunk gets one vector. That vector is, loosely, an average of everything in the chunk. Average five topics together and you get something that scores middling against all five queries and wins none of them.

Large enough to stand alone. The chunk is what gets shown to the model, and often to the user as a citation. "It expires after 30 days" is a perfect single-topic chunk and it is worthless: what expires, and 30 days from when?

Size also decides how much text the model has to read. Chroma's July 2025 report on long inputs tested 18 models of that date, with no RAG pipeline, and found accuracy falling as input grew and a single near-miss passage lowering it. Take the direction and not a number: a big chunk spends the model's attention as well as your precision.

2.How to actually split

Fixed size with overlap is the honest baseline. Cut every N tokens, repeat the tail of each chunk at the start of the next. It works on anything and takes ten lines. Its weakness is that it is blind. It will cut through a sentence, a table or a function body, and a chunk that begins mid-sentence was embedded describing something it does not say.

Structure-aware splitting uses the boundaries the author already put in. Split on headings first, then paragraphs, then sentences, and fall back to a character count only when one paragraph is enormous. This often beats fixed size, and it is cheap to try, for a plain reason: a heading is the author telling you where one topic stops and the next starts, and you paid nothing for that signal. Libraries ship it; Unstructured's by-title strategy, as of October 2026, closes a chunk at each section heading. By default it then merges small adjacent sections, and combine_text_under_n_chars=0 turns that off.

The same rule holds for other formats. Split code on function and class boundaries, markdown on headings, HTML on semantic blocks. Keep tables whole.

Overlap is insurance against boundary cuts, and it is not free. Ten percent overlap means about eleven percent more vectors, and a tendency to return two chunks that are mostly the same text, which wastes slots in your top-k. Ten percent is a rule of thumb, not a measured optimum: a Chroma study from July 2024 found overlap helped in some settings and not others. If you split on structure well you need less, sometimes none.

The header line. A chunk pulled out of a document loses what the document told you about it: its title, its section, its product. Put that back, physically, at the top of the chunk text before you embed it. Instead of the bare sentence Requests submitted within 30 days of purchase are honoured in full., embed Acme Cloud - Billing - Refunds on its own line, followed by that sentence. The vector now knows this is about Acme Cloud refunds, and a query for "Acme refund window" finds it.

A chunk must carry its own context. The embedding can only encode what is in the text, so anything the reader needs from elsewhere in the document has to be copied in.

Acme Cloud - Billing - Refunds
1Requests submitted within 30 days of purchase are honoured in full.2

The evidence is encouraging, with limits. An August 2026 preprint compared eight chunking methods with dense search only and no reranker. A title prefix on 512-token chunks stayed within about a point or two of plain token windows on Recall@100 in most columns, and was sometimes below them, with enriched variants often stronger on top-ten ordering, and no model call needed. Its corpora have titles, it scored whole documents rather than passages, and its winners changed with the embedder. Try this before anything costlier. When a title and heading are not enough, Giving each chunk its context covers the heavier repairs.

# pip install tiktoken
import re, tiktoken

enc = tiktoken.get_encoding("cl100k_base")   # a counter only; use your embedder's if it has one
n_tokens = lambda s: len(enc.encode(s))

def chunk_markdown(doc_title: str, text: str, target=400, overlap=60):
    """Split on headings, then paragraphs. Every chunk carries its header line."""
    chunks, heading = [], ""
    for block in re.split(r"\n(?=#{1,6}\s)", text):          # cut at markdown headings
        first = block.splitlines()[0] if block.strip() else ""
        if first.startswith("#"):                            # assumes no # lines inside code fences
            heading = first.lstrip("# ").strip()
            block = block.split("\n", 1)[1] if "\n" in block else ""   # heading lives in the header
        header = f"{doc_title} - {heading}\n" if heading else f"{doc_title}\n"
        buf, fresh = [], False
        for para in (p.strip() for p in block.split("\n\n") if p.strip()):
            buf.append(para)
            fresh = True
            if n_tokens("\n\n".join(buf)) >= target:
                chunks.append(header + "\n\n".join(buf))
                buf = buf[-1:] if n_tokens(buf[-1]) <= overlap else []   # overlap: last paragraph
                fresh = False
        if buf and fresh:                                    # skip a leftover that is only overlap
            chunks.append(header + "\n\n".join(buf))
    return chunks

The two things this code does that a character-count splitter does not, respecting headings and stamping every chunk with its source, are the cheap first steps. It is not clever, and it does not need to be.

3.Small-to-big, and choosing the size by measuring

The two forces do not have to be settled with one number. Small-to-big retrieval separates them: embed and search small, sharp chunks, but when one wins, return its larger parent, the full section or the chunk plus its neighbours. You match with precision and answer with context.

Cap the expansion. Five hits with 3,000-token parents is 15,000 tokens for the model to read, which brings back the attention cost from section 1. Frameworks ship this pattern, for example LlamaIndex's auto-merging retriever, as of October 2026.

Now the size question, which you should measure. Reuse the 30 labelled queries from Choosing an embedding model, chunk the corpus three ways, and score recall@5 for each. Also count what each size costs. Take 10,000 documents of 2,000 tokens, 10 percent overlap, one 1,024-number float32 vector per chunk (4,096 bytes). The counts are arithmetic. The recall column is invented, to show how to read the table.

The 400-token row wins here. Had 1,000 scored 24, the gap would be one question in thirty, which is noise, and you should take the cheaper index. Chunk count is the lever behind that bill: in the August 2026 preprint, a million web documents produced around five million embeddings, depending on the method.

When you re-chunk, chunk ids change, so label by document and passage rather than chunk id. "Query 7 should retrieve something from refunds.md covering the 30-day window" survives a re-chunk; "query 7 should retrieve doc-14#2" does not. For the same reason the chunker must be deterministic: same text in, same boundaries and ids out. If it is not, Keeping the index fresh without paying twice cannot tell which chunks changed, and every run looks like a full rebuild.

The failure to name: the half table. A pricing table of 900 tokens meets a 400-token splitter. The cut falls after row five, so the second chunk holds rows six to twelve and no header row. A query for the Pro plan seat limit finds it, the model reads "25, 50, 100" with no column names, and it states one of them with confidence. Nothing errored. Keep tables whole up to a size cap, and when one must be split, repeat the header row in every piece.

When not to use this. Skip the splitting work when each record is already a unit: FAQ pairs, glossary entries and product rows are one chunk each. Skip structure-aware splitting on text with no structure, such as transcripts, where fixed size with overlap is fine. Skip small-to-big when your chunks already hold the whole answer, or when parents are large enough to flood the prompt. And if the meaning of a page lives in a chart or a table layout, the page itself can be the unit; Retrieving pages as images covers when.

If recall is poor across all three sizes, size is not your problem. Look at whether your parser is producing garbage; the failure modes and the fix are in Parsing documents before you chunk them. Retrieval cannot recover from text that was already wrong when it arrived.