Tag
experiment-design
- Aug 2, 2025 Confounded Comparisons Between Canary and Baseline A canary differs from its baseline in more than the code it shipped. Traffic mix, cache warmth, host hardware, and routing all shift at once, and a rare-event hypothesis needs far more requests than intuition suggests. observability
- Apr 1, 2024 External Validity and the Limits of What a Benchmark Shows Which conditions a benchmark result actually holds under, how data scale and load level change the regime being measured, and why optimising against a fixed workload stops predicting its own future. performance engineering
- Aug 30, 2023 Pareto Frontiers When No Single Setting Wins Configuration spaces have several objectives that trade against each other; dominance analysis reduces them to a frontier and explains why the frontier, not a score, is the deliverable. performance engineering
- Feb 28, 2023 Variance Reduction by Pairing, Interleaving, and Blocking How separating between-session drift from within-session noise reduces the number of runs a comparison needs, why alternation cancels slow drift but not carryover state, and how to schedule the two versions. performance engineering