Precision and Recall at a Cutoff: Why the Cutoff Changes the Metric

Given a ranked list and a judgment set, a measure has to answer one question: was this list good. The set-based answer is clean and nearly useless. A system that returns every document in the collection achieves total recall and negligible precision; a system that returns one correct document achieves total precision and negligible recall. Neither number changes when the ranking changes, which is a defect, because the ordering is the only thing the retrieval system actually produces.

Cutoff measures exist to make the ordering matter. They truncate the ranked list at a fixed depth and apply a set measure to the prefix. That single change fixes the degenerate cases and introduces a family of new ones, all stemming from the same source: the cutoff is a constant chosen by the evaluator while the number of relevant documents is a property of the topic.

The prefix is the unit of measurement

For a topic with a judgment set and a ranked run, precision at cutoff $k$ is the fraction of the first $k$ results that are judged relevant, and recall at $k$ is the fraction of judged-relevant documents that appear in those first $k$ results.

$$P@k = \frac{|{\text{relevant}} \cap \text{top-}k|}{k}, \qquad R@k = \frac{|{\text{relevant}} \cap \text{top-}k|}{|{\text{relevant}}|}$$

The denominator of precision is fixed at $k$ regardless of how many documents the system returned. A run that returns three documents for a depth-10 evaluation scores at most 0.3, which is the convention: missing results at a depth are treated as nonrelevant, because in a real interface they are absent slots that show nothing to the user. This is why precision at a cutoff can be meaninglessly low for a system that legitimately has fewer than $k$ matches.

The denominator of recall is the number of judged-relevant documents, and that number is topic-specific. It sets a ceiling on how much recall can move as the cutoff grows, and it determines whether a given $k$ is deep or shallow for this topic. A collection whose topics have between two and several hundred relevant documents has no single $k$ that means the same thing everywhere.

Why precision at ten is a coin flip on small topics

The instability of precision at small cutoffs is a variance problem, and it can be quantified without measuring anything.

A top-10 prefix has ten positions. If a system’s true probability of returning a relevant document in any given position is $p$, the number of relevant documents in the prefix behaves like a binomial count, and the standard deviation of $P@10$ is $\sqrt{p(1-p)/10}$ before any per-topic variability in $p$ is even considered. With $p$ around 0.5, one standard deviation is about 0.16. Averaging over many topics shrinks the mean’s standard error, so the mean average of $P@10$ can be reported precisely, but the per-topic values it averages are noisy, and the noise is not the same kind for every topic.

The deeper problem is that $P@10$ is insensitive to anything below rank 10. A system that returns nine relevant documents in ranks 1 through 9 and a nonrelevant document at rank 10 scores the same as one that returns a single relevant document at rank 10 and nine nonrelevant above it, because both contain one relevant document in ten positions. Position within the prefix is invisible to the measure. Reordering relevant documents from ranks 1-3 to ranks 8-10 leaves $P@10$ unchanged, which is precisely the distinction a ranking exists to make.

Aggregating does not repair this. Averaging $P@k$ over topics gives a number with a defensible standard error and an indefensible interpretation: it can be raised by making a system return more documents for topics with few relevant documents, since padding an empty topic with nonrelevant results moves precision toward zero but padding with anything at all cannot move it below zero either. The measure has no floor that reflects the difficulty of the topic.

R-precision anchors the cutoff to the topic

The standard repair is to set the cutoff from the judgments rather than from the interface. For a topic with $R$ judged-relevant documents, R-precision is the precision of the top-$R$ prefix, which because the prefix has exactly $R$ positions is also the recall at depth $R$: the two quantities coincide numerically.

This removes the topic-scaled arbitrariness. Every topic is now evaluated at the depth where recall reaches its maximum possible value, so a system is measured on how many of the topic’s relevant documents it can pack into the number of slots that the topic strictly needs. On a topic with 50 relevant documents, positions 1 through 50 all count; on a topic with 3, only the first 3 positions count, and everything after is ignored.

The cost is that R-precision is harsh and discontinuous. A single nonrelevant document near the top of a topic with three relevant documents removes a full third of the score. R-precision also ignores what happens beyond rank $R$, including the ordering of the relevant documents among themselves, so a run that places its three relevant documents at ranks 1, 2, 3 and one that places them at 1, 40, 60 score identically when $R = 3$. Measures that accumulate credit across ranks, examined in Average Precision and the Mean Over Topics, were designed for exactly this gap.

Precision-recall curves and the eleven-point fossil

The traditional way to sidestep the cutoff choice is to plot precision against recall as the prefix grows, then reduce the curve to a scalar. Precision at each recall level is the observed precision at the first rank where that recall is achieved, with the convention that precision is monotone non-increasing in recall: the interpolated precision at recall $r$ is the maximum precision observed at any recall level at least $r$.

The eleven-point version sampled recall at 0.0, 0.1, …, 1.0 and averaged, a convention from early retrieval evaluation that survives in historical comparisons and in some tooling. The choice of eleven points is a plotting resolution, not a modelling decision, and it biases the average upward relative to the exact area, because sampling the upper envelope of the curve at a coarse grid overweights the high-precision end. Modern reporting has largely moved to measures that average over ranks instead, since a curve averaged over many topics at many recall levels is a large surface to communicate and small differences in it are hard to interpret.

Interpolation itself is a real modelling statement: it says a user is indifferent between a system that achieves recall 0.4 at precision 0.5 and one that achieves recall 0.4 at precision 0.9, provided the second can also reach 0.5 recall at 0.5 precision. That is only true for a user who is willing to keep reading to a recall target. For a user who reads one screen, recall is irrelevant and the cutoff measure is right.

The three regimes that break both measures

Precision and recall at a cutoff fail in identifiable situations, and the failures are worth recognising before choosing a measure.

Topic exhaustion. When a topic has fewer judged-relevant documents than the cutoff, recall saturates at 1 and precision is bounded above by $R/k$. At that point the measure no longer compares systems; it compares the topic’s judgment set against the constant $k$. Reporting a mean over topics where this holds for a subset is reporting a mixture.

Judgment sparsity at depth. Deep cutoffs reach documents that pooling never judged, so the relevant count in the prefix is undercounted and precision is biased downward as $k$ grows. The bias is not uniform, because it depends on how deep each system’s contribution to the pool went. The mechanism is covered in Pooling Depth and the Unjudged Tail.

Ties and truncated runs. Two documents with identical scores have no defined order, so a run that returns them tied has an undefined precision at any cutoff that splits the tie. Scored runs are usually sorted stably by descending score and then by document identifier, which makes the number reproducible without making it meaningful. Runs truncated below $k$ additionally need the padding convention stated explicitly, or the measured value depends on the reader’s assumptions.

None of these are reasons to abandon the measures. They are reasons to state the cutoff, the padding rule, and the tie-breaking rule alongside the number, because the number alone carries none of them.