Power, the Minimum Detectable Effect, and the Cost of Repeated Looks

A significance threshold answers a question nobody asked. Rejecting a null hypothesis at the five percent level says that the observed difference is unlikely under the assumption of no difference; it says nothing about whether the experiment was capable of detecting the difference that mattered. That capability is power, and it is fixed before the data is collected, by the sample size, the variance, and the smallest effect worth acting on.

Power is a property of the design, not the result

Power is the probability that a test rejects the null hypothesis when a specific alternative is true. It depends on the effect size relative to the standard error of the estimate, so for a two-sample comparison of means it is computed from $d \sqrt{n/2}/\sigma$, where $d$ is the difference and $\sigma$ the per-sample standard deviation.

The number that determines whether a benchmark can support any claim is the same quantity rearranged. Fixing the test level at $\alpha$ and the power at $1-\beta$, the required per-arm sample size is approximately

$$n \approx \frac{2 (z_{1-\alpha/2} + z_{1-\beta})^2 \sigma^2}{d^2}$$

from statistics import NormalDist

def runs_per_arm(sigma, effect, alpha=0.05, power=0.8):
    """Runs per arm for a two-sided two-sample comparison.

    sigma is the per-run standard deviation, effect the difference in means
    worth detecting at the stated power.
    """
    z_alpha = NormalDist().inv_cdf(1 - alpha / 2)
    z_beta = NormalDist().inv_cdf(power)
    return 2 * (z_alpha + z_beta) ** 2 * (sigma / effect) ** 2

At the conventional five percent level and eighty percent power, the numerator’s constant is about $2 \times 7.85 \approx 15.7$. The sample size therefore scales as $15.7$ times the squared ratio of the noise to the effect. A harness whose per-run standard deviation is ten times the effect it wants to detect needs about $1{,}570$ runs per arm. That figure, computed before the benchmark is run, is the answer to whether the measurement is worth doing.

The reciprocal is the more useful statement in practice: with $n$ runs per arm, the smallest reliably detectable difference is about $3.96 , \sigma / \sqrt{n}$, since $\sqrt{15.7} \approx 3.96$. The required difference therefore falls as the inverse square root of the run count, and is proportional to the per-run noise. Doubling the run count reduces the detectable effect by a factor of $\sqrt{2}$, which is a slow return. Reducing the variance by a factor of four is worth four times as many runs, and usually much cheaper, which is the argument for the design changes in Variance Reduction by Pairing, Interleaving, and Blocking.

Why power is set before the first run, not after the last

Any quantity computed from the data can be extended after the data exists, but power cannot, because conditioning on the observed effect defeats the purpose. The observed effect is an estimate whose magnitude is inflated when the test rejected; the same selection makes a post-hoc power computed from it a monotone function of the p-value, so it adds no information at all. A test that rejected will always appear to have had adequate power by that computation, and one that did not will always appear to have had too little.

The honest version uses the effect the experiment was designed to detect, chosen from outside the data: the smallest change in the metric that would justify the work of shipping the change. For a hot function, that might be a few percent; for a cold path, no benchmark of that path is needed at all. Choosing it is a decision about the project, not about the statistics, and it must be recorded before the run or the analysis will drift toward whatever was observed.

Peeking inflates the error rate

A running benchmark is tempting to watch. Each refresh is another hypothesis test on accumulating data, and the family of tests has an error rate that compounds.

Under the null hypothesis the sequential estimate behaves like a random walk, and the probability that it crosses a fixed threshold at any point before $n$ samples is higher than the probability that the single test at $n$ samples crosses it. For a two-sided test with a fixed horizon monitored continuously, the crossing probability is roughly twice the nominal rate; for a horizon that is not bounded in advance, continued looking eventually crosses with probability approaching one. A run watched at ten equally spaced points sits between those cases and has an overall error rate several times the nominal five percent.

The mechanism is that a single significant look is enough to stop, so the procedure collects the maximum of many correlated statistics rather than one. The correlation between successive looks reduces the inflation but does not remove it; the runs are not independent, and the boundary is still crossed more often than the nominal rate.

There are two legitimate responses. The first freezes the sample size and analyses once, chosen by the power calculation, which is the classical design. The second is a sequential design with a boundary that is deliberately wider than the fixed-sample threshold, so that the overall error rate is held at the nominal level across all looks.

Sequential boundaries and the price they charge

The standard constructions differ in when they spend the error budget. One family keeps a constant critical value, moderately above the fixed-sample value, at every look; the other declines toward the fixed-sample value as the final look approaches, which makes the early looks very conservative and the final look nearly nominal. In the declining family the first look’s threshold for a handful of looks is more than twice the fixed-sample threshold, so an early decision requires a much larger effect than a final analysis would, while the last look is almost as sensitive as the fixed-sample test.

The gain is in expected sample size. Under a true alternative with a large effect, a design that stops early saves a substantial fraction of the runs; under the null, the run continues to the end. For benchmark comparison where a real regression is suspected and the effect may be large, this trade is good. For detecting small effects, it is bad, because the run almost never stops early and the wider boundary costs power at the end relative to the fixed-sample test.

An alternative is a fixed maximum sample size with early stopping only for a clearly committed decision, and an explicit note that an early stop changes the reported interval’s coverage. Any early-stopping rule that is not accounted for in the reported interval makes the interval too narrow, which is the same failure as peeking without a rule but harder to notice.

The experiment size is a budget, not a target

Each run of a benchmark costs wall-clock time, machine occupancy, and the attention needed to keep the machine in a consistent state. A design that needs fifteen hundred runs per arm at a minute each is a day of machine time per comparison, which is a real constraint on how often regressions can be checked.

This is why the variance reduction work matters more than the sample size arithmetic. A paired design that removes the between-session drift can reduce the required run count by an order of magnitude for the same detectable effect, because it removes a component of variance rather than averaging it down. The same applies to holding the input fixed across versions, to running both versions in the same process where that is meaningful, and to controlling the machine’s frequency state.

The failure mode to avoid is the reverse: reducing the run count to fit a budget while keeping the significance threshold, which silently lowers the power and converts a real regression into a non-significant result. The budget determines the smallest detectable effect; it does not determine whether a test is significant. A design that cannot detect a change worth detecting should report that it cannot, which is more useful than a threshold crossed by luck. The comparison itself, once the sample is fixed, is the subject of Confidence Intervals for the Difference Between Two Benchmark Runtimes, and the methods that avoid distributional assumptions when the sample is small are covered in Distribution-Free Comparison When the Data Is Not Normal.

Reporting what the design could and could not see

A complete report states three things: the effect the design aimed to detect, the number of runs, and the interval on the observed difference. From those, a reader can reconstruct whether a null result means the change was neutral or the experiment was blind.

The distinction between those two cases is the whole point of the analysis. A change with an observed difference near zero and a half-width of half a percent has been shown to be neutral at a useful resolution. The same observed difference with a half-width of eight percent has been shown to be unmeasured. Both produce the same headline if only the point estimate is reported, and the difference between them determines whether the optimisation work should continue or stop.