Inter-Annotator Agreement as a Ceiling on Measured Quality

A quality score is computed by comparing model output to a judgment of what correct output looks like. When that judgment is made by a person, the score is bounded by how consistently people make it. Two annotators reading the same response and applying the same rubric will disagree on some fraction of items, and no automated scorer can be more reliable than the labels it is trained or validated against. The disagreement rate is therefore not a data-quality nuisance. It is the ceiling.

Measuring it correctly is harder than computing the percentage of identical labels, because raw agreement includes agreement that would have happened by chance. Two annotators assigning labels from a set of two will agree half the time even if neither reads the text. Chance correction is what the kappa family of statistics does, and each variant corrects for a different notion of chance and a different scale of label.

Kappa and what it subtracts

For two annotators, nominal labels, and an observed agreement $p_o$, Cohen’s kappa is $(p_o - p_e)/(1 - p_e)$, where $p_e$ is the agreement expected if each annotator’s label distribution were independent. With two labels and a balanced 50/50 split by each annotator, $p_e = 0.5$, so an observed agreement of 0.7 yields $\kappa = 0.4$. The statistic answers: of the agreement that was available beyond chance, what fraction was achieved.

The expected agreement $p_e$ is computed from the marginal distributions of the two annotators separately, not from a shared prior. This makes kappa sensitive to prevalence: when one label is rare, $p_e$ is small, and a modest number of agreements on the rare label produces a large kappa. When both labels are near equally frequent, $p_e$ approaches 0.5 and kappa is harder to move. Two studies with the same raw agreement can therefore report very different kappas, which is why the marginals belong in the report alongside the coefficient.

Fleiss’ kappa extends the same construction to more than two annotators by using the pooled marginal distribution as the chance model and averaging over annotators. It assumes each item is rated by the same number of annotators, drawn from the same population, which is often approximately true and rarely exactly true.

from collections import Counter

def cohen_kappa(a, b):
    assert len(a) == len(b)
    n = len(a)
    observed = sum(1 for x, y in zip(a, b) if x == y) / n
    ca, cb = Counter(a), Counter(b)
    expected = sum((ca[label] / n) * (cb[label] / n) for label in set(a) | set(b))
    if expected == 1.0:
        return 1.0
    return (observed - expected) / (1.0 - expected)

The interpretation bands that circulate, 0.6 as substantial and 0.8 as near-perfect, are conventions from applied statistics, not thresholds with any theoretical standing. What matters for evaluation is the relation between kappa and the metric, not the band.

When the scale is ordinal, the coefficient changes

Quality judgments are frequently ordinal: a response is bad, acceptable, or good, and the distance between bad and acceptable is not the same as between acceptable and good. Nominal kappa treats every disagreement as total, so an annotator calling an item “good” and another calling it “acceptable” counts the same as “good” versus “bad”. That understates agreement on ordinal scales.

Weighted kappa assigns a weight to each cell of the disagreement matrix, typically $1 - (i-j)^2/(k-1)^2$ for a scale with $k$ categories, so adjacent categories cost less than distant ones. The weights are a modeling choice, and linear versus quadratic weighting changes the coefficient substantially on the same data. Reporting a weighted kappa without the weight function is reporting an unlabeled number.

Krippendorff’s alpha generalizes further: any number of annotators, missing data, and any distance metric on the label set, including interval and ratio scales. It is defined as $1 - D_o/D_e$, where $D_o$ is the observed average distance between pairs of judgments of the same item and $D_e$ is the expected distance under random assignment from the pooled distribution. Alpha handles the case where different items were rated by different subsets of annotators, which kappa does not, and it reduces to something close to kappa for nominal data with complete coverage.

The ceiling argument

Suppose the underlying truth is a label per item, annotators reproduce it with some error, and the evaluation set’s gold label is one annotator’s judgment. A scorer validated against that gold label is being asked to reproduce one person’s errors. If two annotators agree on 80 percent of items, then roughly a fifth of the gold labels are judgments a second competent rater would have reversed. A scorer that perfectly matches the gold labels has learned to imitate the specific annotator, including their idiosyncrasies, and its measured accuracy against a third annotator’s labels will be lower.

The ceiling is not exactly the agreement rate. It is the agreement between the gold annotator and the population of competent annotators, and it bounds the expected agreement between any scorer and a fresh annotator. This is the sense in which agreement limits measured quality: a metric that agrees with human judgment on 85 percent of items while humans agree with each other on 80 percent is already at the ceiling, and further metric improvements cannot be validated against that data.

The practical implication is that annotation effort has diminishing returns along the wrong axis. Adding more items to a set with low agreement increases the precision of an estimate of a noisy target. Adding more annotators per item and adjudicating disagreements raises the target’s quality, which is the axis that matters. The standard practice of double-annotation on a subset, with disagreement resolved by discussion, is exactly this: it uses agreement as a filter for which items need attention, and it produces a gold set that is closer to the consensus than any single annotator’s work.

Disagreement is sometimes the correct answer

Not all disagreement indicates error. Some evaluation items are genuinely ambiguous: a response can be factually correct but unhelpful, or helpful but incomplete, and a rubric that forces a single label on such an item manufactures disagreement. When the disagreement rate is high on a specific category of items, the productive response is usually to look at those items and ask whether the rubric is well defined, rather than to assume the annotators are careless.

Reliability statistics computed over a pool that mixes genuinely ambiguous items with clear ones conflate rubric ambiguity with annotator noise. Stratifying the coefficient by category separates them, and a category with low agreement is a signal that the category’s definition needs work. This is the same diagnostic logic that applies to judge agreement in pairwise evaluation, where consistency across presentation orders is checked per stratum rather than in aggregate.

There is a second, subtler issue: annotators are not exchangeable with the population the metric is meant to serve. An annotator pool drawn from one background applies a consistent set of cultural and stylistic expectations, and high agreement within that pool can coexist with systematic divergence from another pool. Agreement measures consistency, not validity. A set of annotators who all share a preference for a particular register will agree closely and will jointly misjudge outputs in another register, and no coefficient computed within the pool detects this.

Chance correction and the base-rate trap

The classic failure of kappa is the high-agreement, low-kappa case. If 95 percent of items are labeled “acceptable” and 5 percent “bad”, and two annotators agree on 90 percent of items, the raw agreement looks strong. But $p_e$ is high, around 0.9, because both annotators overwhelmingly choose the same label, so kappa is near zero. The statistic is reporting that the annotators’ agreement is almost entirely explained by both defaulting to the majority label, which is true and important: the 5 percent of “bad” items, which are the ones the evaluation cares about, are labeled nearly at random.

The opposite trap is a low-prevalence category with a small number of positive items, where a handful of chance agreements produces a large kappa. Both directions argue for reporting the full confusion matrix rather than a single coefficient. The matrix shows where the agreement is and where it is not; the coefficient compresses that into one number and hides the structure that determines whether the metric can be validated.

For evaluation of generated text, the useful summary is usually a pair: raw agreement and a chance-corrected coefficient, with the label distribution stated. The raw agreement tells you how often two humans would produce the same verdict on the same output, which is the number a scorer is trying to match. The corrected coefficient tells you how much of that is real discrimination. Reporting only the second invites the base-rate confusion; reporting only the first overstates reliability on skewed label sets. Neither number substitutes for looking at the items where the annotators split, which is where the metric’s actual boundary is defined.

The agreement rate also sets the scale for any claim about an improvement. If two annotators agree on 82 percent of items, a system that improves its agreement with the gold labels from 74 to 78 percent has moved four points closer to a target that is itself 82 percent reliable, and the remaining gap to 100 is not achievable by better generation. This is the arithmetic behind the observation that benchmark scores saturate: the ceiling is not a property of the model being measured but of the labels it is measured against, and a set with modest agreement cannot distinguish two strong systems no matter how many items are added to it. The confidence interval around each score narrows with more items, which makes the comparison more precise and does not make it more valid.