Tag
evaluation
- Sep 11, 2026 Pooling Depth and the Unjudged Tail How pooled judgments are built from system submissions, why bias grows with pool depth and novelty, and how to measure the reliability of a partial pool. information retrieval
- Nov 24, 2025 How Relevance Judgments Are Encoded in a qrels File The four-field judgment record, the legacy iteration column, and why the choice of a binary or graded scale bounds what any later metric can express. information retrieval
- Nov 19, 2025 Reciprocal Rank and Rank-Biased Precision Two measures built from a model of the user rather than from a fixed cutoff, and the patience parameter that decides where they place their weight. information retrieval
- Oct 8, 2025 Judgment Drift and Assessor Agreement Relevance is not a stable property: label sets shift with the assessor, the interface, and the year. How kappa quantifies that disagreement and how to report metrics in light of it. information retrieval
- Jul 25, 2025 How Many Topics a Benchmark Needs Why the standard error of a mean metric is dominated by topic-to-topic variance, and how resampling shows when a query set is too small to separate two systems. information retrieval
- Mar 25, 2024 Average Precision and the Mean Over Topics How average precision sums precision at every relevant rank, why the mean over topics is a questionable estimator, and what significance testing reveals about it. information retrieval
- Dec 12, 2023 Graded Gains and the NDCG Discount How discounted cumulative gain combines a gain mapping and a rank discount, why the logarithmic discount was chosen, and what normalisation does and does not fix. information retrieval
- Nov 23, 2023 Precision and Recall at a Cutoff: Why the Cutoff Changes the Metric How a rank cutoff turns set-based measures into per-prefix measures, why precision at small k is unstable across topics, and what R-precision fixes. information retrieval
- Sep 6, 2023 Sub-topic Judgments and Diversity Metrics How aspect-level qrels make novelty and coverage measurable, and what the two knobs in alpha-nDCG and intent-aware measures actually control. information retrieval
- Apr 28, 2023 Shadow Traffic: Duplicating Requests to a Candidate Without Serving Them How a serving stack runs a candidate model on a copy of live requests, what the duplication costs at the proxy, and why the comparison it produces is weaker evidence than it looks. inference systems