Context Dilution: When More Retrieved Passages Lower Answer Quality

Retrieval quality is usually discussed as a recall problem: did the right passage make it into the prompt. That framing assumes a monotone relationship between context and answer quality, where more relevant material is better and irrelevant material is neutral. The relationship is not monotone, and irrelevant material is not neutral. A prompt containing the supporting passage plus fourteen topically adjacent passages produces worse answers than the same prompt containing the supporting passage alone, and the degradation is not a rounding error in an evaluation score but a change in which claims the model makes.

The mechanism is competition for attention. Every token in the context competes with every other token through the softmax over attention scores, and the weights sum to one. Adding tokens cannot increase the weight available to the passage that matters; it can only redistribute it. A passage that is on-topic but does not answer the question receives weight because it shares vocabulary with the question, and the weight it receives is weight the supporting passage does not get. The model then produces a plausible synthesis of the whole context rather than an answer drawn from the relevant part, which is exactly the failure that reads as a confident wrong answer.

The competition is arithmetic, not metaphorical

In a single attention head, the output for a position is a weighted sum over all positions, with weights from a softmax. If the supporting passage contributes a fraction p of the total weight and distractor passages contribute the rest, then the representation the model reads is p times the supporting content plus the rest. The softmax’s normalisation is what makes the fraction depend on the number of competitors rather than on the absolute strength of the match: doubling the number of tokens that share the query’s vocabulary roughly halves each one’s share, including the relevant one’s.

Multi-head attention softens but does not remove this. Different heads learn different relations, so some heads may attend sharply to the relevant span while others spread across the context. The output projection combines them, and the combined representation is dominated by whichever heads have the largest norms. There is no mechanism that guarantees the sharp head wins; the model learned a combination during training, on contexts of the lengths and compositions it was trained on, and a context that differs from that distribution shifts the combination.

The practical consequence is that the effect is length-dependent even when the added material is benign. A prompt with five passages and a prompt with fifty, both containing the same relevant passage in the same position, are not the same task. The second is a needle-in-a-haystack task the model may or may not be able to perform, and the ability depends on the model, the position, and the similarity of the haystack to the needle. That is why benchmarks that add irrelevant filler and measure accuracy at increasing lengths show degradation that varies widely across models rather than a uniform curve.

Position is a second axis

The ordering of passages in the prompt is not neutral. Models attend most reliably to the beginning and the end of a long input and least reliably to the middle, an effect documented across model families and across task types. The mechanism is not fully characterised, but the operational fact is stable enough to design around: a relevant passage at position 3 of 40 is used more reliably than the same passage at position 20.

Two explanations are usually offered, and they are not exclusive. Positional encodings make the relative distance between the query and a distant passage large, which flattens the attention distribution over that passage. And the training distribution of long-context data is not uniform, so the model has seen fewer examples that require retrieval from the exact middle of a very long input. Both point the same way: distance from the ends costs reliability.

The design implication is that rank order should determine position, with the highest-ranked passage first and the second-highest last, rather than filling sequentially from the top. That “bookend” arrangement places the two most likely supporting passages at the two positions where attention is most reliable, and it costs nothing. It is a rearrangement of the same tokens and the same budget.

A subtler implication is that adding passages changes the position of every passage after it. Appending a fifteenth passage does not merely add a competitor; if the assembly inserts it before an existing passage, it shifts that passage toward the middle. This is why incremental prompt assembly that appends to the end is safer than assembly that inserts in rank order as results arrive: the latter moves earlier passages on every insertion.

Distractors are worse when they are plausible

The damage from a distractor depends on how well it mimics the supporting passage. A passage about an unrelated topic is easy to discount, because its vocabulary and structure differ from what the question asks for. A passage that discusses the same subsystem, uses the same terminology, and answers a neighbouring question is much harder to discount, and it is exactly what a similarity-based retriever returns: nearest neighbours in embedding space are, by construction, the passages most similar to the query, which correlates with being most similar to the answer.

This is the structural reason retrieval-augmented pipelines are vulnerable to dilution rather than merely to noise. The retriever’s objective is to rank by similarity to the query, and the passages that rank second through fifteenth are the ones that share the most surface and semantic features with the query while not containing the answer. They are adversarial by accident.

Three distractor shapes recur. The version distractor: the same policy from a superseded revision, differing in one clause. The scope distractor: the correct rule for a different configuration, product tier, or jurisdiction. The near-miss distractor: a passage that describes a related mechanism and uses the same nouns, so a model synthesising across the context blends the two. All three are retrieved because they are genuinely similar, and all three are harmful because they are similar enough to be attended to and different enough to be wrong.

Cutting is more effective than adding

Given that context is a competitive resource, the interventions that improve answer quality are mostly subtractions, and they act at different stages.

The first is a relevance threshold rather than a fixed k. Retrieve a candidate list, then include only passages whose score clears a cutoff calibrated on the encoder and metric in use. A fixed k guarantees a fixed number of competitors regardless of whether the corpus contains that many relevant passages; for a query the corpus answers thinly, the fixed k is a guarantee of dilution. The threshold has to be calibrated per encoder and per corpus, for the reasons described in Similarity Metrics and the Normalisation They Require, and it should be validated against a labelled sample rather than chosen for a round number.

The second is deduplication by content. Overlapping chunks and multiple formats of the same document produce repeated text, and repetition consumes budget and attention while adding no information. Near-duplicate detection at assembly time, by shingle overlap or by embedding distance within the retrieved set, removes the repeats before they compete.

The third is reranking before assembly. A cross-encoder distinguishes “answers this question” from “is about this topic” in a way that a dual encoder cannot, so it is the tool that removes plausible distractors rather than merely reordering them. The number of passages that survive reranking is the number that enter the prompt, which makes reranker depth and prompt size the same decision seen from two ends, treated in Cross-Encoder Reranking.

The fourth is structural: give each passage an explicit boundary and an identifier, and instruct the model to answer from the passages rather than from the context as a whole. Delimiters do not eliminate competition, but they make the spans distinguishable, which helps both attention and attribution. The same identifiers are what make citation possible, as described in Span Attribution.

def assemble(question, ranked, budget, threshold, count_tokens, near_dup):
    # count_tokens and near_dup are supplied by the caller: a token counter and
    # a near-duplicate test over two strings
    chosen, used = [], 0
    for chunk in ranked:
        if chunk.score < threshold:
            break                       # below cutoff: stop, do not pad with weak hits
        if any(near_dup(chunk.text, c.text) for c in chosen):
            continue
        n = count_tokens(chunk.text)
        if used + n > budget:
            continue                    # try a smaller later chunk rather than truncate
        chosen.append(chunk)
        used += n
    # bookend: best first, second best last, the rest in between
    if len(chosen) > 2:
        chosen = [chosen[0]] + chosen[2:] + [chosen[1]]
    return chosen

The continue rather than break at the budget check is a deliberate difference from the earlier loop: with a threshold in place, a large chunk that does not fit can be skipped in favour of smaller ones that do, because the ranking already established that everything remaining clears the cutoff.

Measuring dilution rather than assuming it

The effect is measurable without inventing benchmarks, and the measurement is a controlled comparison on a fixed evaluation set. For each question with a known supporting passage, assemble prompts at increasing context sizes: the supporting passage alone, then with the next four retrieved passages, then with the next sixteen. Score the answers against the reference. The curve that results is the dilution curve for that model, corpus, and assembly policy, and its shape is what determines the operating point.

Two properties of that curve are worth expecting. It is usually non-monotone: quality rises from one passage to a few, because additional context sometimes supplies a qualifier the answer needs, and then falls as competitors accumulate. And it is model-specific, so a value measured for one model does not transfer to another, which means the curve has to be re-measured whenever the generator changes.

The measurement also exposes the interaction with position. Running the same comparison with the supporting passage placed first versus placed in the middle separates the effect of context size from the effect of position, and the two are often confounded in informal evaluation: adding passages pushes the supporting one toward the middle, so a measured drop may be attributable to either.

The limitation of the whole approach is that it optimises a proxy. Reference-based scoring rewards matching the expected answer, and a model that answers from a distractor in a way that is internally consistent but wrong scores as wrong, which is correct. A model that refuses to answer because the context is contradictory scores as wrong too, though refusal may be the better behaviour. The choice of how refusals are scored changes the optimal context size, and that choice is a policy decision about the product rather than a property of the retrieval system.