Performance Engineering · Cluster
Measurement
Benchmark design and statistical validity.
- 01 Warmup Transients and Detecting Steady State in Timed Loops Why the first iterations of a timed loop measure a different program, which subsystems are still converging, and how to locate the transition into steady state rather than assuming it.
- 02 Clock Sources and the Resolution Floor of a Timing Call How monotonic clocks, TSC reads, and syscall-based time differ in cost and granularity, why a measured interval is a difference of two uncertain readings, and how the floor limits what a benchmark can resolve.
- 03 Measurement Overhead and the Code the Compiler Deletes How dead-code elimination, loop unrolling, and constant folding silently remove the work a benchmark claims to time, and which structural defences keep the timed region intact.
- 04 Quantiles from Finite Samples and Their Confidence Bands What a p99 computed from a few hundred observations actually estimates, how the order statistic maps to a binomial interval, and why tail claims need sample sizes that frequently do not exist.
- 05 Confidence Intervals for the Difference Between Two Benchmark Runtimes Why two overlapping intervals can still hide a real difference, how the standard error of a difference is computed from paired or independent samples, and what a ratio estimate needs before it is trustworthy.
- 06 Power, the Minimum Detectable Effect, and the Cost of Repeated Looks How sample size is determined by the smallest effect worth detecting rather than by convenience, why peeking at a running experiment inflates false positives, and what sequential designs cost in exchange.
- 07 Distribution-Free Comparison When the Data Is Not Normal What rank tests actually test, how permutation and bootstrap procedures produce intervals on a benchmark difference without assuming a shape, and where each method loses power.
- 08 Variance Reduction by Pairing, Interleaving, and Blocking How separating between-session drift from within-session noise reduces the number of runs a comparison needs, why alternation cancels slow drift but not carryover state, and how to schedule the two versions.
- 09 Averaging Ratios and the Geometric Mean in Composite Scores Why the arithmetic mean of normalised ratios depends on the reference machine, when the geometric mean is the right summary, and where the harmonic mean is forced by the arithmetic of rates.
- 10 External Validity and the Limits of What a Benchmark Shows Which conditions a benchmark result actually holds under, how data scale and load level change the regime being measured, and why optimising against a fixed workload stops predicting its own future.