Pooling Depth and the Unjudged Tail

Judging a collection exhaustively is not possible. A collection of a few million documents with a few dozen topics would require hundreds of millions of assessments, and the assessors do not exist. The standard alternative is to judge only the documents that some system returned near the top of its ranking, and to treat everything else as though it had been judged nonrelevant. This is pooling, and its consequences infect every metric computed from a pooled collection.

The mechanism is simple: collect the top $d$ results for every submitted run, take the union, remove duplicates, and hand the result to assessors. Depth $d$ is the pool depth, usually 100 for the classic TREC-era collections, sometimes smaller when budget is tight. Documents not in the union are unjudged, and the collection documentation does not record their absence explicitly: the qrels file simply has no row for them.

Why pooling works at all

The justification is an unstated assumption that has held well enough in practice to become a convention. A system that is genuinely better tends to retrieve the same documents that other good systems retrieve, because relevance is a property of the document and the query rather than of the system. A new system’s novel documents, the ones no other system found, are more likely to be noise than treasure, and the chance that one of them is relevant and highly ranked is small when the pool is deep.

The argument is empirical and its strength depends on the pool depth relative to the rank at which the evaluated system places its results. At depth 100, a novel relevant document at rank 20 of a system that no other system resembles is unjudged, scored as irrelevant, and the system loses a contribution it should have received. At depth 10, the same loss would be far more likely, because ten positions is a small enough window that a genuinely different system has a real chance of putting relevant documents there that nobody else did.

Bias from pooling is therefore not random noise that averages out over topics. It is one-directional: unjudged documents are scored as nonrelevant, and some of them are relevant, so every pooled collection understates the performance of systems that contributed less to the pool. The size of the understatement depends on the system and on the topic, which is worse than a uniform offset, because it can reorder systems.

The pool is a function of the submissions

A collection’s qrels are not fixed by the collection. They are fixed by the set of runs submitted in the year the collection was built. A system that differs from that set in its approach, or one that retrieves documents from a part of the corpus other systems rarely reach, is evaluated against a judgment set that was constructed without regard to it.

This matters for two ordinary situations. Benchmark continuation: a new system is scored against a collection assembled years ago, and the pool contains documents retrieved by a retrieval paradigm that has since been abandoned. If the new system’s approach reaches different material, its score is depressed for reasons unrelated to quality. Harvesting: a system that uses a different tokenisation or a different language model of the query will surface documents that the pool’s creators never saw, and their absence from the judgments looks like failure.

The literature on incremental and low-cost assessment exists because of this. Rather than re-judging the whole collection, additional methods judge only the documents a new run places above a threshold that the existing pool did not cover, folding the new judgments into a revised qrels file. The revised file is comparable to the old one only if the evaluation tooling accounts for the difference in pool coverage, which it cannot do automatically.

Depth versus cost, and the measurable trade

Pool depth is the only lever an evaluation campaign controls directly, and its cost is linear in the number of runs multiplied by the depth. Doubling the depth roughly doubles the assessment workload, so campaigns balance against a budget.

The relationship between depth and missed relevant documents can be estimated after the fact with a residual measure. Take the runs used to build the pool, remove one run, and evaluate it against the pool built from the others. If the removed run is scored much lower than when the full pool is used, the pool depth is insufficient for that run: the judgments it would have needed came from itself, and removing it exposed the gap. Aggregating this over all submitted runs gives an estimate of how much the collection depends on any single system, which is exactly the dependence that pool bias creates.

A complementary technique is to judge a sample of the unjudged documents, drawn from the tail of the runs, at a known sampling rate. If a fraction of the sample turns out relevant, the missed count can be estimated and compared to the number of judged relevant documents. The estimate is crude, because the sample only covers documents that some run returned, but it converts an invisible bias into a number.

Scoring a partial pool without pretending

Given unjudged documents, the analysis has to choose a treatment, and the choices produce different numbers from identical inputs. The three that appear in tooling:

Treat as nonrelevant. The default. Simple, reproducible, and biased downward against systems whose results were not judged. Correct only when the pool is complete at the evaluated depth.

Condense. Remove unjudged documents from the ranked list and evaluate the remaining sequence at their new, higher positions. This assumes unjudged documents carry no information and that the user would have skipped them. It biases upward, because the removed documents may have been nonrelevant and their removal improves the measured positions of the kept ones.

Restrict the evaluated depth. Evaluate only to the depth at which every returned document is judged. Honest, but it caps the cutoff and can make deep measures meaningless.

None of the three is universally right, and mixing them across a comparison invalidates the comparison. The minimum requirement is that the treatment be fixed before the analysis and reported with it. A more useful practice is to report the per-run unjudged fraction within the evaluated depth, so a reader can see whether the measure had full information. A run whose top 10 contains no unjudged documents needs no caveat; one whose top 10 is half unjudged does.

Why rank-sensitivity makes the tail matter more than it looks

The intuitive response to pool bias is that the tail is worth little, since discounts shrink with depth, so missing documents there cannot matter. This is wrong for the measures that accumulate credit across the whole list, and it is wrong in a specific way.

Average precision and NDCG count every relevant document in the judged depth. If a system retrieves 80 relevant documents and the pool only contains judgments for 60 of them, the measured AP is roughly proportional to 60/80 of the true value, regardless of where the missing 20 sat. The relative loss is about the same as the fraction missed. Since pool depth determines which documents are judged, and deeper-ranked relevant documents are the ones most likely to be missed, systems that retrieve many relevant documents deep in their lists are penalised more than systems that retrieve few. The bias scales with recall, which is the opposite of what a fair measure should do.

Shallow-cutoff measures are less exposed. RBP at $p = 0.5$ places negligible weight below rank 20, where the pool is nearly complete; NDCG at 10 likewise. The measure’s cutoff and the pool’s depth are the pair that determines whether a comparison is trustworthy, and the rule of thumb is that the evaluated cutoff should sit no deeper than the pool depth. Evaluated cutoffs deeper than the pool depth are measuring the pool, not the system.

What can be verified before publishing a number

A few checks distinguish a collection that supports the comparison from one that does not.

• The per-run fraction of unjudged documents within the evaluated cutoff. Near zero is a green light; a double-digit percentage means the number needs a caveat.
• The number of submitted runs contributing to the pool, relative to the number being evaluated. A pool built from three runs supports three runs poorly.
• The distribution of judged-relevant counts per topic. Topics with very low counts make deep truncation measures unstable, as discussed in Precision and Recall at a Cutoff.
• Whether the evaluated runs were part of the pool. If they were not, the pool’s coverage of their results is an open question, and the removal test above estimates the answer.

The deeper point is that pooled evaluation measures a system’s agreement with the pool’s consensus as much as its relevance. That is acceptable when systems are diverse and the pool is deep, and misleading when a new approach reaches material the consensus never judged. Recognising that is the difference between reporting a number and claiming a fact.