How Relevance Judgments Are Encoded in a qrels File

A retrieval system cannot be scored against the truth. The collection contains no field marking which documents answer a query, and no amount of algorithmic work recovers that field, because the property being measured is a relation between a document and a person’s information need. Evaluation therefore substitutes a sample: a set of people look at a set of documents for a set of topics and record a verdict for each pair. Those verdicts, serialized into a plain text file called a qrels file, become the fixed point against which every subsequent measurement is defined.

Treating that file as a data format rather than a formality explains most of the strange properties of retrieval metrics. The format is small, the conventions attached to it are not, and the interpretation chosen at write time constrains every metric that can later be computed from it.

A judgment is a pair, and the pair is unordered

The basic record has four whitespace-separated fields:

301 0 FR940104-0-00001 2
301 0 FR940104-0-00007 0
302 0 WSJ910101-0004 1

The fields are the topic identifier, an iteration marker, the document identifier, and the relevance label. Nothing about the line carries a rank, a score, or a timestamp. The file states which documents were judged and what the verdict was; the order in which the systems under test happened to return them is a separate artifact, held in a run file, and the two are joined at measurement time by matching document identifiers.

The second field is a fossil. Topic sets built for relevance-feedback experiments could be judged repeatedly: a system retrieved a first pass, the user marked documents relevant, and the query was reissued with those documents as feedback, producing a second round of judgments for the same topic. The iteration column distinguished those rounds. Collections assembled after that experimental style fell out of use write 0 there, and current tools ignore the field when matching, so a document judged in any iteration counts for the topic. A parser that keys on the first three fields is doing what the tools do; a parser that keys only on (topic, document) and discards duplicate rounds is doing what most people want.

Because the record identifies a pair, the same document may legitimately carry different labels under different topics. A judgment is never a property of a document alone.

Reading the file without corrupting it

Two properties of the format deserve enforcement rather than trust. Identifiers must not contain whitespace, since whitespace is the only separator; and the label must be read as an integer, including when it is negative. TREC collections have used -1 for documents that an assessor marked as not yet judged, and a reader that clamps labels at zero silently turns a placeholder into a nonrelevant verdict.

A second trap is duplication. A qrels file assembled by merging two assessors, or by concatenating judgments made in successive years, can contain the same pair twice with different labels. Nothing in the format resolves the conflict. The reader has to pick a policy explicitly: take the maximum, take the first occurrence, average the labels, or refuse the file and report the collision. Silent last-write-wins is the default of a naive dictionary assignment and is the worst of the four, because the result then depends on line order, which is not part of the format’s semantics.

def load_qrels(path):
    """Parse a TREC qrels file into {topic: {docid: label}}."""
    judgments = {}
    for line in open(path, encoding="ascii"):
        parts = line.split()
        if not parts or parts[0].startswith("#"):
            continue
        topic, _iteration, docid, label = parts[:4]
        per_topic = judgments.setdefault(topic, {})
        rel = int(label)
        if docid in per_topic and per_topic[docid] != rel:
            # Keep the stricter verdict rather than depending on line order.
            rel = max(rel, per_topic[docid])
        per_topic[docid] = rel
    return judgments

That function returns a partial map. The word partial matters more than anything else in this article: the absence of a pair means not judged, not not relevant. Every downstream metric collapses the distinction, and the collapse is discussed in Pooling Depth and the Unjudged Tail.

The label scale bounds the metric

The most consequential decision in building a judgment set is how many distinct labels the assessors may use, and every collection makes it before any metric is chosen.

A binary scale records one verdict: relevant or not. It is cheap, it is reproducible between assessors because the decision boundary is coarse, and it makes recall well defined, since the denominator is simply the number of judged-relevant documents. It also throws away information that no later processing recovers. Once the two documents that answer a query are both labelled 1, a metric cannot distinguish a ranking that puts the direct answer first from one that puts a tangential mention first.

A graded scale keeps that ordering. Scales of three and four levels are the common choices, usually with the top level reserved for documents that answer the need by themselves. The labels are ordinal, not interval: the step from 0 to 1 need not equal the step from 1 to 2, and a metric that sums labels has implicitly declared that it does. This is not a flaw in the format, since the format stores integers and nothing else, but it is a modelling assumption that the metric makes on the format’s behalf. The assumption is examined where gains are turned into numbers, in Graded Gains and the NDCG Discount.

The scale also fixes the dynamic range. With four levels and an unbounded ranked list, a graded measure has more room to separate systems than any binary measure derived from the same judgments, because the binary measure sees at most two values per document. Conversely, a binary collection fed to a graded metric does not become more informative: gain becomes 1 for every relevant document, and the only remaining signal is position.

Recall’s denominator is an artifact of the file

Set-based recall compares the retrieved relevant documents against all relevant documents. Since the file does not contain all relevant documents, only those that someone judged, recall computed from qrels is really recall over the judged subset. The denominator is a count of rows in the file.

The consequence is that recall values are comparable only within one collection, and not always then. Two topics in the same collection can have wildly different numbers of judged-relevant documents, in part because the topic is broader, in part because the pooling process happened to surface more of them. A system that retrieves ten relevant documents scores higher recall on the topic with twelve known relevant documents than on the topic with three hundred. Averaging recall over topics therefore mixes quantities with different ceilings, which is why the mean is usually reported alongside per-topic values rather than on its own.

There is no format-level fix for this. The file records judgments, not sampling probabilities, so it cannot say what fraction of the relevant material was found. A collection built by pooling has a known relationship between pool depth and the chance of missing relevant documents, but that relationship lives in the collection’s documentation, not in the qrels file.

What the record cannot carry

The four fields are deliberately minimal, and the omissions define the boundary of what evaluation on a qrels file can ask.

There is no field for why a document was judged relevant, so a collection cannot distinguish a document that answers the question from one that merely discusses the topic. There is no field for which aspect of an ambiguous query a document covers, so nothing computed from a plain qrels file can reward a result set that covers several interpretations, which is the subject of Sub-topic Judgments and Diversity Metrics. There is no assessor identifier in the standard layout, so agreement between assessors has to be tracked in a parallel file, and there is no confidence field, so a marginal verdict is stored as if it were certain.

Systems that need any of these extend the record rather than replace it. Aspect-aware collections append the aspect to the document identifier or ship a second file keyed on the pair. Agreement studies keep one qrels per assessor and compare them afterwards. In both cases the original four-field file remains the join key, and the extension is documentation that travels with the collection.

The practical rule that follows: read the collection’s documentation before the qrels, because the file will parse perfectly while meaning something narrower than expected. A parse that succeeds is not a judgment set that supports the comparison being attempted.