Reciprocal Rank and Rank-Biased Precision

The measures examined so far take a list and a set of judgments and assign a number by arithmetic on positions. Precision at a cutoff prices every position up to $k$ equally and everything after at zero. Average precision prices position $i$ at roughly $1/i$ but only counts documents that happen to be relevant. Neither corresponds to a stated model of how far a user reads and what happens when they stop.

Two measures make the model explicit, and both are easier to reason about because of it. Reciprocal rank assumes a user is looking for one document and stops when they find it. Rank-biased precision assumes a user reads a geometrically distributed number of results and stops when they are satisfied. Each reduces to a formula with a single psychological parameter, and in each case the parameter, not the arithmetic, is what determines how the measure treats deep lists.

Reciprocal rank and the single-answer topic

For a topic where one document is the answer, the natural measure is the rank at which that document appears, inverted so that higher is better:

$$\mathrm{RR} = \frac{1}{r}$$

where $r$ is the position of the first relevant document. The full answer first scores 1.0, second scores 0.5, tenth scores 0.1, and a list with no relevant document in the judged depth scores 0. Mean reciprocal rank over topics is the mean of these values.

MRR is the right measure for a class of tasks and the wrong measure for everything else, and the distinction is entirely about the user’s stopping rule. In question answering, in navigational search, and in entity lookup, the user wants a single target and the list is a guess ordered by likelihood. Everything below the first relevant result is invisible, so the measure is indifferent to it. This produces a failure on topics with several valid answers: a system that returns three legitimate answers and puts them at ranks 1 through 3 scores 1.0, and a system that returns a better ordering of three hundred results with the first relevant at rank 2 scores 0.5. The measure is not wrong; it is answering a question about a task where only one result matters.

MRR is also dominated by the reciprocal’s steep early decay. The difference between rank 1 and rank 2 is 0.5, the difference between rank 11 and rank 12 is 0.006. Averaging over topics, a single topic where the answer moved from rank 1 to rank 10 shifts the mean by up to 0.09, which is larger than the difference between many competing systems. That is a property of the scale rather than a defect in the estimate, and it means that reporting MRR alongside average precision is informative in a way that reporting MRR alone is not: a system that does well on both is finding single answers early and finding many of them.

Rank-biased precision and the geometric stopping rule

Rank-biased precision starts from a different premise. A user examines results in order and continues with probability $p$, independently at each step. The probability that the user reaches position $i$ is then $p^{i-1}$, and the measure is the expected gain from the document at each position, with nonrelevant documents contributing nothing:

$$\mathrm{RBP} = (1 - p)\sum_{i=1}^{n} r_i , p^{i-1}$$

The scale factor $1-p$ normalises the geometric series, so a list of all-relevant documents scores 1.0 regardless of $p$ and the result stays in the unit interval. The residual term is the probability mass the user never reached, $p^{n}$, which is exactly the chance that they would have gone further and found something the run did not contain.

The parameter $p$ is the measure’s entire personality. At $p = 0.5$ the user is expected to examine two results, and the effective depth is short. At $p = 0.8$ the expectation is five results. At $p = 0.95$ it is twenty. At any value the measure has a well-defined depth of about $1/(1-p)$ positions, which is a concrete statement about the user rather than a number chosen for convenience.

Two consequences distinguish RBP from AP and NDCG. First, it never divides by the count of relevant documents, so it does not inherit the topic-varying denominator that makes topic-level average precision hard to average. Second, its value is interpretable as a probability: the chance that a user with this patience, reading this list, encounters a relevant document before giving up. A system scoring 0.6 under $p = 0.8$ is one where a user with five-result patience finds something useful 60 percent of the time.

The measure also exposes a trade-off that others hide. Raising $p$ deepens the evaluation and rewards runs that are good in the tail; lowering it concentrates on the head. Two systems compared at $p = 0.5$ and at $p = 0.95$ can swap places, and the swap is not an inconsistency but a statement that the systems have different strengths at different depths. Reporting RBP at one $p$ is reporting a preference.

Handling graded gains and the unjudged tail

RBP generalises past binary relevance by substituting the label’s gain for $r_i$. The same caveats as with NDCG apply: the gain function is a modelling choice, and the scale factor no longer guarantees a maximum of 1.0 unless the maximum possible gain is 1. The residual $p^n$ becomes more interesting in this setting, because a graded run that stops early leaves more potential gain unexplored.

Unjudged documents present the measure’s sharpest assumption. RBP’s usual formulation requires that a document not in the judgment set be treated as nonrelevant, which is defensible only when the pool is deep enough that the unjudged tail is genuinely unlikely to contain relevant material. Moffat and Zobel argue the point from the direction of pool construction: the evaluation is sound when every document that a reasonable set of systems returned was judged, because then the unjudged space is unexplored territory that no system claimed. Against a shallow pool, RBP inherits the same bias as every other measure and shows it more visibly, because the geometric weighting concentrates mass on the head where the pool is densest but the residual term explicitly counts the unexplored remainder.

A second formulation scores documents by their judged status directly, using the discrete probability distribution over the number of documents read. Both give the same ranking of systems when the judgments are complete, and they differ in how the unjudged case is handled; the choice should be stated.

Which model fits which task

The two measures are not competitors so much as descriptions of different users, and the choice between them should follow from what the system is for.

Measure	User model	Effective depth	Penalises	Natural task
Reciprocal rank	Stops at first relevant result	Rank of first hit	Delaying the single answer	Question answering, navigation
RBP, $p = 0.5$	Reads about 2 results	Very shallow	Anything below rank 3	Mobile, single-snippet interfaces
RBP, $p = 0.8$	Reads about 5 results	First screen	Poor head ordering	Web search first page
RBP, $p = 0.95$	Reads about 20 results	Two screens	Poor mid-list ordering	Exploratory search with browsing

A system optimised for one row of that table is not optimised for another, and a benchmark that reports a single measure implicitly selects one. The failure mode to watch is optimising against a measure whose user model contradicts the product: a system tuned on RBP at $p = 0.5$ learns to put one good result first and does not learn to order anything else, which is fine for a snippet interface and poor for a results page that a user scrolls.

The general form: measures as expected utility

Both measures fall out of a single construction. Choose a distribution over the depth at which the user stops, and choose what the user gains from a document at each depth. The measure is the expectation of the gain under that distribution. Reciprocal rank corresponds to a stopping rule that places all its mass on the position of the first relevant document; RBP corresponds to a geometric stopping rule independent of relevance; average precision corresponds to a stopping rule that depends on the relevant count and on the reader’s reactions. Once the construction is stated, the differences between the measures are differences between stopping rules, not differences between formulas.

That reframing is useful for the harder cases. When the interface shows snippets of unequal length, the cost of reading a result varies with position and with the document’s snippet, and no standard measure accounts for it. When the user can reformulate the query cheaply, the value of a deep result set shrinks, and the effective $p$ should fall. The parameter is not a free variable to tune until a system wins; it is a claim about behaviour, and it should be set from evidence about behaviour where such evidence exists, and stated as an assumption where it does not.