Chunk Boundaries Decide What Can Be Retrieved

Retrieval is a competition between units. The index stores some set of spans, the encoder compresses each span into one vector, and the query picks the vectors closest to it. Every property that makes a span a good index entry is in tension with every property that makes it a good prompt entry. Index entries want to be narrow, so that one vector represents one idea and the nearest-neighbour search can discriminate between them. Prompt entries want to be self-contained, so that a reader who sees only that span understands it. A chunk is where those two requirements are forced into the same object, and the boundary is the compromise.

The tension is measurable. Take a passage of 400 tokens that explains a caching policy: the first 40 tokens name the policy and state what it does, the middle 300 give the algorithm, and the last 60 discuss when to disable it. Split at 256 tokens and the first chunk contains the naming and most of the algorithm; the second contains the tail of the algorithm plus the caveat, and it opens with a pronoun whose referent is in the other chunk. A query asking “when should the cache be disabled” matches the second chunk on the caveat text but the encoder also sees 190 tokens of algorithm that dilute the vector. A query asking “what does the cache do” matches the first chunk cleanly. The same document, split differently, produces different recall for different questions. There is no split that is optimal for both.

Boundaries are placed by four different rules

The simplest rule is a fixed token count. Emit spans of exactly N tokens, where N is chosen to fit several chunks into the context budget. This rule is fast, needs no parser, and produces uniformly sized vectors, which matters for indexes that store vectors in fixed-width blocks. It also cuts sentences in half at every boundary that does not happen to fall between them, and a sentence cut mid-clause is not a sentence to either the encoder or the reader. A variant splits on the nearest whitespace or punctuation before the limit, which avoids mid-word cuts but still permits mid-sentence ones.

The second rule splits on sentence boundaries. Segment the text into sentences, accumulate sentences until the token count would exceed N, emit, and start again. This guarantees that no sentence is broken and that chunks end at a natural pause. It does not guarantee that the chunk is about one thing: a paragraph of four sentences about three different failure modes becomes one vector averaging all three. Sentence segmentation itself is not trivial; the boundary rules in Unicode Text Annex #29 handle the cases a naive split on periods gets wrong, such as abbreviations, decimals, and ellipses.

The third rule follows document structure. Markdown headings, HTML elements, and code fences are explicit statements about where a unit of meaning ends. Splitting on a level-two heading yields chunks that correspond to what the author considered a topic, and prepending the heading path to the chunk text gives the encoder context it would otherwise lack: a section titled “Retry policy” under “Connection handling” means something different from a section with the same title under “Batch jobs”. Structural chunking fails when the author’s sections are much longer or shorter than the budget, so implementations usually treat structure as a first pass and the token limit as a second.

The fourth rule is semantic: embed candidate sentences, compute similarity between adjacent ones, and place a boundary where similarity drops. It costs an embedding call per sentence at index time and produces chunks of unpredictable size. The gain over structural splitting is real for unstructured prose and small for documents that already have headings, and the cost is paid once per document rather than once per query, which is the only reason it is affordable at all.

def split_by_structure(text, max_tokens, count_tokens):
    # sections are (heading_path, body); heading_path is a list of titles
    # parse_sections and split_sentences are the document parser and the
    # segmenter; both are supplied by the caller, not shown here
    chunks = []
    for path, body in parse_sections(text):
        prefix = " > ".join(path)
        for para in body.split("\n\n"):
            n = count_tokens(prefix) + count_tokens(para)
            if n <= max_tokens:
                chunks.append((prefix, para))
                continue
            # oversized paragraph: fall back to sentence accumulation
            buf, size = [], count_tokens(prefix)
            for sent in split_sentences(para):
                s = count_tokens(sent)
                if size + s > max_tokens and buf:
                    chunks.append((prefix, " ".join(buf)))
                    buf, size = [], count_tokens(prefix)
                buf.append(sent)
                size += s
            if buf:
                chunks.append((prefix, " ".join(buf)))
    return chunks

The prefix in that sketch is doing real work. A chunk of 200 tokens that begins with “Retry policy > Backoff” costs four extra tokens and removes an ambiguity that the encoder would otherwise have to infer from vocabulary alone.

Overlap buys recall and pays in duplicates

The standard remedy for boundary loss is overlap: consecutive chunks share a span of tokens, typically between 10 and 25 percent of the chunk size. The reasoning is direct. If the answer sentence sits at the boundary, one of the overlapping chunks contains it whole, with enough surrounding text to be interpretable.

Overlap has three costs, and they are not symmetric.

Index size grows by the overlap fraction. A 20 percent overlap on a 500-token chunk means the corpus is stored 1.2 times, and every derived structure grows with it: vectors, posting lists, and the memory the index occupies at serving time. For a corpus of a million chunks that is an extra two hundred thousand entries, which is a real cost in an index held in RAM.

Duplicates enter the prompt. When two overlapping chunks both match, both are retrieved, and the prompt contains the same paragraph twice. The generative model has no reliable way to know that the repetition is an artefact, and repetition in the context tends to be echoed in the output. The deduplication in the prompt assembly step is not optional; without it, overlap converts directly into wasted context budget.

Ranking gets noisier. Two chunks that share 20 percent of their text produce similar vectors, so the score gap between them is small and the order in which they appear is decided by noise. If the pipeline passes the top k to a reranker, the reranker sees near-duplicate candidates and spends capacity distinguishing them instead of distinguishing relevant from irrelevant.

A middle position exists: overlap by sentence rather than by token, so that the shared span is a whole sentence or two. The shared text is then semantically complete, and a reader encountering it twice sees a repeated sentence rather than a truncated fragment. This is the usual configuration for structural chunking, where the natural overlap unit is the last paragraph of the previous section.

Size trades precision against sufficiency

Chunk size sets two properties at once, and they pull in opposite directions.

Small chunks produce sharp vectors. A 128-token chunk about one thing has a vector close to that thing, so nearest-neighbour search discriminates well and the top result is likely to be on-topic. But a 128-token chunk often lacks the antecedent, the units, or the qualifier that make the statement usable. The model receives a true but incomplete passage and must either guess the missing context or state that it cannot answer.

Large chunks produce sufficient context and blunt vectors. A 2,000-token chunk contains the answer and everything needed to interpret it, but its single vector is an average over many topics, so it competes poorly against a 200-token chunk that is entirely on-topic. The retrieval step then returns the wrong large chunk, or returns the right one at a rank too low to survive truncation.

The common resolution is to decouple the two units. Index small spans, retrieve by them, then expand each hit to its parent section before inserting it into the prompt. The vector stays sharp because it represents the small span; the prompt stays sufficient because it receives the parent. The cost is bookkeeping: the index must store a parent pointer per chunk, and expansion must respect the token budget, which is why the prompt assembly loop counts tokens rather than chunks. A variant indexes several granularities at once and merges results, which raises recall and also raises the number of near-duplicate candidates, pushing more work onto the reranking stage described in Cross-Encoder Reranking.

The boundary is fixed at index time

The decision that makes chunking expensive is that it is made before any query exists, and changing it invalidates everything downstream. Chunk boundaries determine the vectors, the vectors determine the index contents, and the index is what the serving path reads. Rechunking a corpus means re-embedding it, which for a large corpus is hours of accelerator time, and rebuilding the index, which may mean taking it offline or building a shadow copy.

That cost pushes design toward chunking rules that are stable under corpus growth. A rule keyed to document structure is stable: adding a section does not move the boundaries of other sections. A rule keyed to a fixed token offset is not: inserting a paragraph near the top shifts every subsequent boundary, so a corpus that is edited frequently re-chunks almost everywhere on every update. This is one argument for structural splitting that has nothing to do with retrieval quality, and it interacts directly with how the index is maintained incrementally, covered in Incremental Index Maintenance.

Two boundary cases deserve explicit handling. Tables and code blocks are not prose and do not survive sentence splitting; a table row split from its header is uninterpretable, and a function split from its signature is worse. Structural chunking should treat them as atomic units even when they exceed the token limit, and the prompt budget should accommodate the overflow rather than cutting them. The other case is the document shorter than one chunk. It should be indexed whole, without a synthetic split, and its metadata should record that it is a whole document so that expansion does not attempt to fetch a parent that does not exist.