Tag
relevance-judgments
- Sep 11, 2026 Pooling Depth and the Unjudged Tail How pooled judgments are built from system submissions, why bias grows with pool depth and novelty, and how to measure the reliability of a partial pool. information retrieval
- Nov 24, 2025 How Relevance Judgments Are Encoded in a qrels File The four-field judgment record, the legacy iteration column, and why the choice of a binary or graded scale bounds what any later metric can express. information retrieval
- Oct 8, 2025 Judgment Drift and Assessor Agreement Relevance is not a stable property: label sets shift with the assessor, the interface, and the year. How kappa quantifies that disagreement and how to report metrics in light of it. information retrieval
- Mar 25, 2024 Average Precision and the Mean Over Topics How average precision sums precision at every relevant rank, why the mean over topics is a questionable estimator, and what significance testing reveals about it. information retrieval
- Nov 23, 2023 Precision and Recall at a Cutoff: Why the Cutoff Changes the Metric How a rank cutoff turns set-based measures into per-prefix measures, why precision at small k is unstable across topics, and what R-precision fixes. information retrieval