How Many Topics a Benchmark Needs

A benchmark reports a mean over topics. The mean is a sample statistic computed from whatever topics happened to be written, and the question that follows immediately is how much of the number is signal. Changing the topic set changes the mean even when the systems do not change, and a comparison that looks decisive on one set can reverse on another. The size of that instability is estimable, and the estimate usually shows that benchmarks are smaller than their reported precision suggests.

There is no closed form for “the number of topics needed”, because it depends on how heterogeneous the topics are, how much systems differ, and how the metric responds to that difference. What can be done is to measure the variance empirically on an existing collection and reason from it. That is the whole method, and it is short enough to apply to any collection with a run.

The variance that matters is between topics

Metrics are not independent across topics. The difficulty of a topic, the number of relevant documents, and the quality of the judgment set vary from one topic to the next, and those differences dominate the differences between systems.

The practical consequence is that a single number cannot be quoted with a useful confidence interval unless the between-topic spread is known. Consider two systems evaluated on 50 topics. Their mean average precisions differ by a small amount, and every per-topic difference is small, but 30 topics favour one system and 20 favour the other. The mean says one is better; the per-topic pattern says the answer depends on which topics matter. Reporting only the mean makes this invisible.

The standard way to expose it is a paired resampling test. Resample the topic set with replacement, compute the mean for both systems on the resampled set, and record which system is ahead. Repeat many times, and the fraction of resamples in which the second system leads is an estimate of the probability that the observed ordering would reverse under a different, equally valid topic sample. That fraction is directly interpretable, unlike a p-value from an inappropriate normality assumption.

import random

def paired_bootstrap_diff(metric_a, metric_b, trials=10000, seed=0):
    """metric_* : per-topic scores for two systems, aligned by topic."""
    rng = random.Random(seed)
    n = len(metric_a)
    idx = range(n)
    wins = 0
    diffs = []
    for _ in range(trials):
        sample = [rng.choice(idx) for _ in range(n)]
        mean_a = sum(metric_a[i] for i in sample) / n
        mean_b = sum(metric_b[i] for i in sample) / n
        diffs.append(mean_a - mean_b)
        if mean_a < mean_b:
            wins += 1
    diffs.sort()
    lo, hi = diffs[int(0.025 * trials)], diffs[int(0.975 * trials)]
    # wins/trials estimates P(A loses) under topic resampling.
    return diffs[trials // 2], (lo, hi), wins / trials

The function returns the median difference, a 95 percent interval on it, and the fraction of resamples that reverse the ordering. An interval that contains zero and a reversal rate in the tens of percent mean the comparison is not settled, regardless of the size of the two means.

Resampling topics treats the topics as exchangeable, which is an assumption and not a fact. If the query set is deliberately stratified, with navigational, informational, and transactional topics in fixed proportions, then resampling uniformly changes the proportions and inflates the variance relative to the real sampling scheme. The fix is to resample within strata, at a cost of a few more lines and a much more honest interval.

Why 50 topics is a defensible floor and not a target

Classic evaluation campaigns settled on sets of roughly 50 topics per track, chosen because assessment cost scales linearly and 50 was affordable. The choice has been examined in reverse: how often would the conclusions change if the topic set were different.

The findings that recur across such analyses are consistent in direction. The number of topic sets on which the ranking of two systems is stable rises with the size of the topic set, and the improvement needed to be reliably detected falls as the set grows. For differences of a few percentage points in mean average precision, sets of 50 topics frequently fail to separate systems, while sets of several hundred are more likely to. The exact figures depend on the metric and on how similar the systems are, and quoting a specific figure from a specific study would be quoting that study’s systems.

The practical reading is that a 50-topic benchmark is adequate for detecting large differences and inadequate for detecting small ones. A system that improves by a wide margin will show it; a system that improves by a narrow margin needs either many more topics or an evaluation method that reduces variance. That second option is the one that matters for a system whose entire value proposition is a narrow improvement, and it is why paired designs and low-variance measures have been pursued.

Separating rescoring from re-evaluation

The cost of a big topic set is not in the arithmetic. Computing metrics over a thousand runs and a thousand topics is trivial on any modern machine. The cost is in judgment: every topic needs a pool and assessors, and the assessment budget is the binding constraint.

That is why the practical advice is to extract more conclusions per judgment rather than to judge more. Three techniques reduce the variance of a comparison without adding topics:

Paired evaluation. Score both systems on the same topics and compare per-topic differences, not the means independently. Most of a topic’s difficulty is shared by both systems and cancels in the difference, so the variance of the difference is far smaller than the variance of either mean. This is nearly free and is the highest-value change a comparison can make.

Low-variance measures. A measure whose per-topic values are tightly clustered produces a more precise mean for the same topic count. Rank-biased precision at a moderate patience parameter has this property relative to average precision, because its values are bounded and do not depend on a per-topic relevant count in the denominator. The trade in user-model realism is discussed in Reciprocal Rank and Rank-Biased Precision.

Reuse of judgments. Running a new system against an existing collection costs no assessment at all, provided the pool coverage discussed in Pooling Depth and the Unjudged Tail is adequate. The savings are large and the bias is real, and both belong in the report.

The relationship between metric variance and topic selection

A topic set is not a uniform sample from a population, because the population of information needs is not enumerable and the topics are written by people with a purpose. Two consequences follow.

First, topic sets drift toward the concerns of their year. A collection built for a web-search track contains topics whose vocabulary matches the web; a collection built for a patent corpus contains topics phrased in patent language. A system tuned on one and shipped to the other meets a different distribution, and the benchmark’s mean transfers only if the distributions are close. This is a bias problem rather than a variance problem, and no amount of resampling reveals it.

Second, some topics are more informative than others. A topic where every submitted system scores near 1.0 contributes almost nothing to distinguishing systems, and a topic where every system scores near zero contributes nothing either; both add variance-free noise to the mean and consume judgment budget. Filtering out topics where all systems cluster at the extremes raises the discriminating power of the remaining set at the cost of a topic set that no longer reflects the natural mixture. The choice should be stated, because a filtered topic set produces a mean that is not comparable to an unfiltered one.

What a sample-size claim can honestly say

Given a collection and two runs, the bootstrap above answers a specific question: on topic sets drawn from this one, how often does the ordering hold. That is a statement about the topic set, not about the world, and the two are different.

An honest report therefore has three parts rather than one. It states the mean and the topic count. It states the interval or the reversal rate from a paired resampling, so the reader can see the sampling error. And it states whether the evaluated runs contributed to the judgment pool, so the reader can judge the pool bias. Those three sentences fit in a paragraph and do more for the credibility of a benchmark result than any amount of additional measurement on the same topics.

The direction of the remaining improvement is also clear from this account. Variance falls with the square root of the topic count, so quadrupling the topics halves the interval. Between-topic variance falls much faster when comparisons are paired on the same topics, so pairing is cheaper than expansion. And the largest errors in small benchmarks are not sampling errors at all but pool and judgment bias, which no increase in topic count removes.