Tag
statistical-significance
- Jul 17, 2026 How Many Requests a Canary Gate Needs Before Its Verdict Means Anything The sample-size arithmetic behind canary health checks, why rare failures and tail latencies need orders of magnitude more traffic, and how sequential testing changes the threshold. distributed systems
- Aug 2, 2025 Confounded Comparisons Between Canary and Baseline A canary differs from its baseline in more than the code it shipped. Traffic mix, cache warmth, host hardware, and routing all shift at once, and a rare-event hypothesis needs far more requests than intuition suggests. observability
- May 13, 2025 Confidence Intervals for Accuracy on Small Evaluation Sets A score on a few hundred items is a sample from a binomial, and the interval around it is usually wider than the differences being reported. How to compute it and when the independence assumption fails. inference systems
- Aug 15, 2023 Comparing Two Classifiers with Paired Tests on the Same Rows How to test whether an observed difference in error is larger than fold noise, why the resampled t-test fails, and what the corrected alternatives assume. machine learning