Judgment Drift and Assessor Agreement

A judgment set is written as though relevance were a property of a document. The assessor reads the query, reads the document, and records a verdict, and the file stores that verdict as an integer with no indication that another competent assessor might have written a different one. Measurement built on those integers inherits the assumption of stability, and the assumption is false in ways that have a specific structure.

Disagreement between assessors is not noise around a true label. It is a distribution of legitimate readings, and its width is a fact about the task. Collections that report metrics without mentioning it are reporting digits whose last few places are not measuring retrieval quality at all.

Two ways to disagree

Assessor disagreement arises from two different sources, and they need different treatments.

The first is ambiguity in the topic statement. A query like “capital gains tax” can be answered by a statute, an explanation of the concept, a rate table, or a news article about a proposed change. Assessors reading the same document in light of the same topic can differ on whether it answers the need because they differ on which need the topic names.

The second is threshold placement. Even with an agreed interpretation, the boundary between “relevant” and “marginally relevant” is a judgement call, and two assessors will not place it in the same place. On a graded scale, this produces disagreement concentrated on adjacent labels, which is a much less serious problem than disagreement across the relevant/nonrelevant line.

The distinction matters because adjacent-label disagreement barely affects a binary measure and heavily affects a graded one with an exponential gain function, while cross-line disagreement affects both. The agreement statistics described below do not distinguish the two cases, so they must be read alongside a confusion matrix of label pairs.

Kappa and what it subtracts

The standard statistic for agreement between two annotators on nominal labels is Cohen’s kappa, defined as the observed agreement corrected by the agreement expected from the marginals:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

where $p_o$ is the fraction of items on which the annotators agree and $p_e$ is the agreement expected if each annotator’s label distribution were independent of the other’s. The subtraction of $p_e$ is the point: two assessors who label 90 percent of documents relevant will agree 82 percent of the time by chance alone, so raw agreement overstates consistency badly when the labels are skewed.

Kappa’s interpretation as a scale from poor to almost perfect is widely quoted and weakly grounded. The value depends on the prevalence of each label, so a kappa computed on a collection where one label dominates is not comparable to a kappa computed on a balanced sample. Its ceiling is below one whenever the marginals differ, which means two assessors applying identical criteria with different label rates produce a kappa below one. And it is computed over nominal categories, so a graded scale has to be treated either as nominal, discarding the ordering, or with a weighted variant that incorporates the distance between labels.

For grading this is a real gap. A weighted kappa that gives partial credit for adjacent labels is the right choice when the measure of interest is graded, and the weight choice is itself a modelling decision. The more informative presentation for a collection is a full confusion matrix over the label pairs, from which both the agreement statistic and the direction of disagreement can be read.

How much disagreement there actually is

The empirical literature on assessment found stable results across several campaigns. Complete re-assessment of a collection, with the same topics and the same documents but different assessors, produces agreement figures that look unimpressive for an exact match, and materially better for a coarser binary decision.

The critical finding is not the level of agreement but its effect on system comparisons. Re-evaluating the same submitted runs against two independently judged sets produces means that differ, often by amounts comparable to the differences between systems, while the ordering of systems is largely preserved. That asymmetry is the practical case for pooled evaluation: absolute scores from a collection are conditional on who judged it, but relative comparisons survive better than absolute ones.

The preservation is not universal. Systems that are close in performance can swap places when the judgments change, and systems with different characteristics swap more often than systems built on the same principles. The effect is another argument for the variance-aware reporting described in How Many Topics a Benchmark Needs: a difference that survives one judgment set and disappears under another was never large enough to act on.

The assessment interface is part of the measurement

Judgments are produced through a tool, and the tool’s design changes the judgments. Three design choices have documented effects on label distributions.

Presentation order. Assessors presented with documents in a fixed order drift toward a criterion and then apply it; presentations that interleave topics, or that show a document alongside the topic again, disrupt that drift. Collecting judgments topic by topic concentrates all of one topic’s assessments in one person’s attention, which increases internal consistency but reduces independence from whatever criterion that person established early.

Grading rubric. Whether the rubric distinguishes topical relevance from answering the need determines whether the top label means anything. A rubric with an explicit example per label produces tighter agreement than one that describes the levels abstractly. The gain functions in Graded Gains and the NDCG Discount become interpretable only if the labels were defined this way.

Available information. An assessor shown the query and the document judges the document. An assessor shown the snippet judges the snippet, which is what the user actually saw, and produces a different label set. Neither is wrong; they measure different things, and a collection built with snippet-level assessment should not be used to evaluate full-document retrieval without saying so.

Cheap assessment and its cost

Because judgment is the dominant expense in building a collection, a body of work examines how far assessment can be reduced before the resulting metrics stop being trustworthy. Two directions have been studied.

Sampling documents rather than judging all of them. Judging a random subset of the pool and estimating the metric from the sample turns evaluation into survey estimation. The metric becomes an unbiased estimate of the full-pool value with a known variance, which is more defensible than judging the head fully and inferring the tail, provided the sampling probabilities are recorded and used in the estimator. The variance is large for measures that depend on deep ranks, because a large share of the contributions comes from a small share of documents.

Crowdsourcing judgments. Outsourced assessment is cheaper per judgment and noisier per judgment. The standard mitigations are redundant labelling of each pair with a majority vote, and calibration questions with known answers used to filter workers. Majority voting reduces the variance of a label but shifts it toward the mode of the worker population, which is not the same as a careful assessor’s verdict; the two collections are not interchangeable even when their agreement with each other looks respectable.

The general conclusion that has held up is that a collection can be made cheaper by judging fewer documents deeply rather than more documents carelessly, and that partial assessment must be paired with an estimator that accounts for what was skipped. Discarding the distinction between judged-nonrelevant and never-judged is exactly the error that a sampling framework is designed to avoid.

Reporting practice that survives the disagreement

Three sentences make a metric report adequate to everything in this section, and none of them requires additional assessment.

State the labeling scale and the rubric that produced it, so a reader knows what the integers mean and can map them to their own interpretation. State the pool depth and the fraction of judged documents at the evaluated cutoff, so the reader knows whether the denominator was complete. And state whether the comparison was tested against topic resampling, so the reader knows whether the difference exceeded the noise in the collection.

When a collection has been re-judged, the stronger report includes both numbers, the original and the re-judged, with the note that the ordering was preserved or not. That is more informative than the usually quoted single figure, and it is the only form in which the sensitivity of the conclusion to the judgments is visible to the reader rather than known only to the people who built the collection.