Inference Systems · Cluster
Evaluating Outputs
Quality measurement for generated text.
- 01 Clipped N-Gram Precision and the Brevity Penalty How BLEU converts n-gram overlap into a single number, why counts are clipped against the reference, and where the brevity penalty stops correcting for short output.
- 02 Recall-Oriented Summary Scoring with ROUGE Why summarization metrics divide by reference length rather than output length, how longest-common-subsequence scoring differs from n-gram counts, and what the recall direction lets a system get away with.
- 03 Why Perplexity Is Not Comparable Across Tokenizers Perplexity measures average surprise per token, so the vocabulary decides the unit. Converting to bits per byte removes the unit but not every confound.
- 04 Answer Normalization and the Fragility of Exact Match Exact-match scoring turns a string comparison into a quality verdict. The normalization pipeline in front of the comparison decides how much of the score is measurement.
- 05 Position Bias in Pairwise Preference Judging When a judge compares two responses, the order they appear in changes the verdict. Measuring the effect requires swapping the pair and reading the disagreement rate.
- 06 Confidence Intervals for Accuracy on Small Evaluation Sets A score on a few hundred items is a sample from a binomial, and the interval around it is usually wider than the differences being reported. How to compute it and when the independence assumption fails.
- 07 Inter-Annotator Agreement as a Ceiling on Measured Quality If two careful human raters disagree on a third of items, no scorer can exceed that agreement. Cohen kappa, Krippendorff alpha, and why the ceiling is not a target.
- 08 Detecting Benchmark Contamination from the Model Side A score on a public test set measures memorization as well as skill. N-gram overlap against training data, perturbation tests, and the limits of each detector.
- 09 The Unbiased Pass@k Estimator for Code Generation Sampling k completions and checking whether any passes is a biased estimate of pass@k. The combinatorial correction, why it is needed, and how variance scales with the sample.
- 10 Decomposing Generated Text into Atomic Claims for Factuality Scoring Sentence-level factual accuracy conflates several errors into one label. Splitting output into self-contained claims changes both the denominator and the failure modes.