Information Retrieval · Cluster
Retrieval Evaluation
Relevance judgments and ranking metrics.
- 01 How Relevance Judgments Are Encoded in a qrels File The four-field judgment record, the legacy iteration column, and why the choice of a binary or graded scale bounds what any later metric can express.
- 02 Precision and Recall at a Cutoff: Why the Cutoff Changes the Metric How a rank cutoff turns set-based measures into per-prefix measures, why precision at small k is unstable across topics, and what R-precision fixes.
- 03 Average Precision and the Mean Over Topics How average precision sums precision at every relevant rank, why the mean over topics is a questionable estimator, and what significance testing reveals about it.
- 04 Graded Gains and the NDCG Discount How discounted cumulative gain combines a gain mapping and a rank discount, why the logarithmic discount was chosen, and what normalisation does and does not fix.
- 05 Reciprocal Rank and Rank-Biased Precision Two measures built from a model of the user rather than from a fixed cutoff, and the patience parameter that decides where they place their weight.
- 06 Pooling Depth and the Unjudged Tail How pooled judgments are built from system submissions, why bias grows with pool depth and novelty, and how to measure the reliability of a partial pool.
- 07 How Many Topics a Benchmark Needs Why the standard error of a mean metric is dominated by topic-to-topic variance, and how resampling shows when a query set is too small to separate two systems.
- 08 Interleaving as an Online Evaluation Method Team-draft and balanced interleaving merge two rankings into one list, so that clicks on the result can be used to compare systems without relevance judgments.
- 09 Judgment Drift and Assessor Agreement Relevance is not a stable property: label sets shift with the assessor, the interface, and the year. How kappa quantifies that disagreement and how to report metrics in light of it.
- 10 Sub-topic Judgments and Diversity Metrics How aspect-level qrels make novelty and coverage measurable, and what the two knobs in alpha-nDCG and intent-aware measures actually control.