Sub-topic Judgments and Diversity Metrics

Every measure considered so far assumes that a document’s value is determined by its own label and its position. That assumption fails for a large class of queries. The query “jaguar” has an animal sense, a car sense, and a software sense, and a list of ten documents about the car is worth less to a general audience than a list that covers all three. The query has a single topic statement and several underlying needs, and no per-document judgment can express the difference.

Aspect-level collection design is the standard answer. Instead of one judgment per document, the collection carries a set of sub-topics derived from the query, and each document is judged with respect to each sub-topic it addresses. An evaluation measure then operates over a set of independent rankings, one per sub-topic, and combines them into a score that rewards coverage and penalises repetition.

How aspect-level judgments are stored

The record gains a dimension. A plain qrels file has (topic, document, label); an aspect-level collection has (topic, aspect, document, label), or equivalently a document identifier that encodes the aspect, or a separate file listing document-aspect pairs alongside the document-level labels.

The derivation of aspects is the design decision that constrains everything downstream. Two families exist. Intent-based aspects come from users, harvested from query logs or from a taxonomy of interpretations for ambiguous queries. They capture the senses that real users have in mind, which makes the resulting measure a reasonable proxy for user satisfaction, and they require log data or manual enumeration that not every evaluation has. Content-based aspects come from the documents themselves, often by clustering the results for the query and treating each cluster as an aspect. They are cheap to derive and reproducible, and they produce aspects that reflect the corpus rather than the users: a cluster around a rare sense will be treated as equally important as one around the dominant sense.

The choice changes what the metric rewards. Intent-based aspects reward covering the senses people actually mean. Content-based aspects reward returning documents from more of the corpus’s divisions, which may or may not correspond to what a user needs. The collection documentation must say which was used, because the two are not comparable.

From a ranking to a set of rankings

Given aspects, a ranked list is reinterpreted as several lists. For each aspect, filter the original ranking to the documents that address that aspect, preserving order. A twenty-document run over a query with four aspects becomes four lists of varying lengths. Run a standard measure on each and combine.

Alpha-nDCG does this with a parameter that controls the combination. For each aspect $a$:

$$\mathrm{nDCG}_a@k = \frac{1}{\mathrm{IDCG}a@k}\sum{i=1}^{k}\frac{g_a(d_i)}{\log_2(i+1)}$$

and the combination weights each aspect by the probability that a user intended it:

$$\alpha\text{-nDCG}@k = \sum_{a} P(a \mid q),\mathrm{nDCG}_a@k$$

The probabilities $P(a \mid q)$ come from the aspect derivation: for intent-based aspects, the share of queries or users with that intent; for content-based aspects, a uniform distribution is the usual fallback. Setting them uniform makes every aspect count equally, which is a strong claim about a query whose senses are wildly unequal in frequency.

The measure carries a second parameter, conventionally also called alpha, applied to the gain function to reward redundancy differently. In the formulation that names the measure, the gain of a document for an aspect is reduced if a similar document has already been seen at a higher rank, so that a list of ten near-duplicates about the car scores far below a list that covers the car with three documents and moves on. The similarity function is either a fixed threshold on a fingerprint comparison or a graded overlap, and its parameters are part of the measure’s configuration in exactly the way the discount function was in Graded Gains and the NDCG Discount.

What the two knobs control

The parameters are easy to confuse because both are called alpha in the literature and they do different things.

The aspect weighting parameter decides how the per-aspect scores combine. When it puts all mass on one aspect, the measure reduces to an ordinary graded measure on that aspect. When it spreads uniformly, it rewards coverage. A system can be compared to itself under different weightings to see whether its advantage comes from breadth or from depth on the dominant aspect.

The redundancy parameter decides how quickly repeat coverage of an aspect stops paying. At zero, a document is worth its full gain regardless of what preceded it, and the measure becomes redundant with per-aspect grading. At high values, the second document on the same aspect is worth almost nothing, and the measure becomes a coverage count. The setting determines whether near-duplicate results are penalised heavily, which is the behaviour the measure exists to produce.

Neither parameter is estimated from data in standard use. Both are set by the evaluator, usually at a value that makes the measure behave sensibly on a few example lists, and reported with the result. That is a weaker foundation than the discount function’s, which at least has a geometric argument behind it, and it means the numbers produced are only comparable at the same settings.

Intent-aware recall and the coverage framing

A different family of measures treats the problem as set coverage rather than ranked gain. Intent-aware recall at cutoff $k$ computes, for each aspect, the fraction of that aspect’s relevant documents in the top $k$, then averages over aspects with the intent probabilities as weights. A variant asks a simpler question: how many distinct aspects are represented in the top $k$ at all, ignoring how many documents each contributes.

The coverage framing is more interpretable and less sensitive to configuration. It answers the question a product owner asks, which is whether the first screen shows the user that several interpretations exist. It is also cruder: it does not distinguish a list where one aspect is represented by a single document from one where it is represented by several, and it does not price the position at which the coverage appears.

For sub-topic queries the two framings should be reported together. A system that covers every aspect with one mediocre document per aspect and a system that covers two aspects thoroughly and ignores the rest have similar scores under some settings and very different scores under others, and the pair of numbers distinguishes them where either alone does not.

The failure modes that come from the judgments

Aspect-level evaluation inherits every problem of ordinary pooled evaluation and adds three.

Sparse aspect judgments. If the pool is thin, aspects that few systems addressed are represented by only the documents that happened to be judged, and a system that returns good material for a neglected aspect is scored against a judgment set that contains one document for it. The per-aspect ideal list is then computed from a nearly empty set, which makes the ratio unstable in the same way that R-precision is unstable on topics with few relevant documents, discussed in Precision and Recall at a Cutoff.

Aspect exhaustion at the cutoff. When an aspect has fewer judged documents than $k$, every system that finds them all scores the same, and the measure stops discriminating. On a query with four aspects and a shallow aspect, a large share of the total score becomes a constant. Aggregated over queries, the constant depresses means and compresses differences, which is the opposite of what a diversity measure is introduced to achieve.

Aspect proliferation. A collection with many fine-grained aspects gives every system credit for something, so all scores rise and the ranking becomes uninformative. A collection with very few aspects collapses toward a single-aspect measure. The granularity of the derivation is therefore a collection-level parameter that affects comparability, and a result computed against aspects at one granularity cannot be compared to a result against a coarser derivation.

Where diversity evaluation stops being useful

The measures above assume that the value of coverage is a property of the query. For a genuinely ambiguous query, that holds. For a query with one answer that happens to have many related documents, rewarding coverage rewards a list that answers the question less directly.

This is why diversity tracks in evaluation campaigns paired aspect-level judgments with explicit graded labels for the primary interpretation: a document that answers the likely need should outscore a document that covers a secondary aspect, and the combination has to encode that ordering. A diversity measure that treats all aspects as equal on a query dominated by one interpretation will prefer a worse list, and the mistake is invisible in the aggregate score.

The final caution is about metric gaming. A list that returns one document per aspect at the bottom of the cutoff can score well on coverage while being useless to a user who reads only the first result. Position and aspect weighting interact, and the interaction should be checked by looking at the per-aspect ranking rather than at the aggregate. The aggregate is a summary; the per-aspect lists are the measurement.