Why a Larger Context Window Does Not Remove the Need for an Index

A context window is a property of one forward pass. A corpus is a property of an organisation, and the two have never been the same size. A model that accepts 128,000 tokens holds about 250 pages of text at 500 tokens per page, which is enough capacity to feel like a replacement for search until the corpus is measured against it. Twenty thousand pages of product documentation is 10 million tokens, roughly 78 full windows. A decade of support tickets, a source tree, or a statute book runs larger again. The ratio between corpus and window, not the window in isolation, is what decides whether a selection step is needed.

That ratio moves in one direction over time. Windows have grown by roughly an order of magnitude per model generation, but corpora grow with every commit, ticket, and revision, and they never shrink. A window large enough for today’s documentation is not large enough for the documentation plus the changelog plus the incident archive plus the three previous versions still under support. Retrieval is the mechanism that decouples prompt size from corpus size, and it does so by paying a fixed cost per query instead of a cost proportional to everything ever written.

The four stages between a query and a prompt

A retrieval pipeline is short enough to describe completely. The corpus is segmented into chunks, each chunk is encoded into a vector or an inverted-index posting list, the encodings are stored in a structure that supports nearest-neighbour search, and at query time the same encoder maps the question into the same space, the structure returns the k closest chunks, and those chunks are concatenated into the prompt ahead of the question. Only the last stage involves the generative model at all; the first three are ordinary information retrieval, and they run on CPUs or on a separate accelerator pool.

The pipeline has one hard requirement that is easy to violate: the query encoder and the document encoder must be the same function, or at least two functions trained to produce comparable vectors. A model trained to embed passages and a model trained to embed questions produce vectors in spaces that are related but not identical, and mixing them degrades ranking in ways that look like bad luck rather than a bug. The standard arrangement, described in the dense passage retrieval work, uses a single dual-encoder with two input formats.

def build_prompt(question, hits, budget_tokens, count_tokens):
    # hits are (chunk_id, text, score), already ranked
    parts, used, seen = [], 0, set()
    for chunk_id, text, score in hits:
        if chunk_id in seen:          # overlapping chunks can duplicate text
            continue
        n = count_tokens(text)
        if used + n > budget_tokens:  # stop rather than truncate mid-sentence
            break
        seen.add(chunk_id)
        parts.append(text)
        used += n
    body = "\n\n".join(parts)
    return f"{body}\n\nQuestion: {question}\nAnswer:"

Two details in that sketch matter more than they appear. The budget is checked against the assembled text rather than against a count of chunks, because chunks differ in length and a fixed k produces wildly variable prompt sizes. Duplicate suppression exists because overlapping chunking, which is the common remedy for boundary loss, will return the same sentence twice when two neighbours both match.

Prefill is charged per token, and twice

The cost of a long prompt is not just the cost of reading it. Two quantities grow with input length, and they grow at different rates.

The first is the attention computation. Self-attention builds a score matrix of shape n by n for each head, so the attention term in the prefill FLOP count grows with the square of the sequence length, while the term for the feed-forward and projection weights grows linearly. At short prompts the weight term dominates and the quadratic term is invisible. At the tens of thousands of tokens that long-context models advertise, the quadratic term is the reason prefill latency curves upward rather than staying flat per token.

The second is the key-value cache. Every token in the prompt stores one key and one value vector per layer for the duration of the generation. For a 32-layer model with 4,096-wide key and value projections in 16-bit precision, one token costs 2 x 32 x 4,096 x 2 bytes, which is 512 KiB. A 100,000-token prompt therefore pins about 50 GiB of accelerator memory before a single output token is produced. Grouped-query attention cuts this in proportion to the ratio of query heads to key-value heads, but the linear-in-tokens shape remains. A prompt of 100,000 tokens is not 780 times more expensive than a prompt of 128 tokens; it is more expensive in a different category, because it consumes capacity that would otherwise hold other requests.

Retrieval replaces a variable, corpus-sized input with a bounded one. A budget of 4,000 tokens of retrieved context costs 2 GiB of cache at the geometry above, which fits alongside other work. The saving is not a constant factor; it is the difference between a request that can be batched and one that runs alone.

Selection fails in ways that loading everything does not

The honest argument for stuffing the whole corpus into the window is that it cannot miss. Retrieval can. If the chunk containing the answer is not in the top k, the model answers from parametric memory or refuses, and no amount of context budget recovers the passage. This failure is silent: the prompt looks well-formed, the retrieved passages are plausible, and the answer is wrong in a way that reads fluently.

Three mechanisms produce that miss. The chunk may be cut at a boundary that separates the question’s phrasing from the answer, which is why chunk size and overlap are a design decision rather than a preprocessing detail, treated in Chunk Boundaries Decide What Can Be Retrieved. The index may be stale relative to the corpus, so a document added after the last build is invisible. Or the embedding may simply place the relevant passage far from the query, because a short question and a long explanatory paragraph are not distributionally similar even when they are semantically related. Approximate index structures add a fourth: the search itself is allowed to miss neighbours in exchange for speed, so recall is a tuning parameter rather than a guarantee, as described in Approximate Nearest Neighbour Indexes.

Stuffing has a failure mode of its own, and it is less obvious. Adding passages that are topically related but irrelevant lowers answer quality rather than leaving it unchanged; the effect is documented well enough that it has a name, context dilution, and it is the subject of Context Dilution. Position within the prompt matters too. Models attend most reliably to material at the beginning and end of a long input and least reliably to material in the middle, so a relevant passage placed at position 40,000 of a 100,000-token prompt is not equivalent to the same passage at position 200. A large window does not make every position equally usable.

The cases where loading everything wins

Retrieval is not universally correct, and there are configurations where a single long prompt is the better engineering choice.

The first is a corpus small enough to fit with room to spare. If the entire knowledge base is 30,000 tokens, an index adds a build step, a serving dependency, and a failure mode in exchange for nothing. The threshold is not the window size but the fraction of it the corpus occupies; once the corpus exceeds roughly a quarter of the window, there is no space left for the conversation, the instructions, and the output.

The second is a task that requires the whole document at once. Summarising a contract, diffing two versions of a specification, or answering a question whose evidence is spread across sections defeats chunk-level retrieval, because no single chunk contains the answer. The remedy is document-level selection followed by full inclusion, which is retrieval with k equal to one and a chunk equal to a document.

The third is a workload with no stable query distribution. An index is built around an assumption about how questions will be phrased and which parts of the corpus they will target. When every query is a one-off exploration over a modest corpus, the amortisation never happens: the encoder still runs, the index still needs maintenance, and the fixed cost per query is paid for a benefit that a full-context call would have delivered anyway.

Outside those cases, the index is not an optimisation of the context window. It is the component that decides what the window contains, and every later stage in this cluster, from chunking through filtering to attribution, exists because that decision is approximate.