Tag
evaluation-protocol
- Apr 22, 2026 Inter-Annotator Agreement as a Ceiling on Measured Quality If two careful human raters disagree on a third of items, no scorer can exceed that agreement. Cohen kappa, Krippendorff alpha, and why the ceiling is not a target. inference systems
- Feb 19, 2026 Position Bias in Pairwise Preference Judging When a judge compares two responses, the order they appear in changes the verdict. Measuring the effect requires swapping the pair and reading the disagreement rate. inference systems
- Jun 18, 2025 Decomposing Generated Text into Atomic Claims for Factuality Scoring Sentence-level factual accuracy conflates several errors into one label. Splitting output into self-contained claims changes both the denominator and the failure modes. inference systems
- May 13, 2025 Confidence Intervals for Accuracy on Small Evaluation Sets A score on a few hundred items is a sample from a binomial, and the interval around it is usually wider than the differences being reported. How to compute it and when the independence assumption fails. inference systems