Measurement Overhead and the Code the Compiler Deletes

A benchmark that computes a result nobody reads has written a program whose observable behaviour is the same as one that does nothing, and an optimising compiler is entitled to notice. The result is not a wrong measurement of the intended work but an accurate measurement of nothing, reported with the full confidence of a clean distribution. The defences against this are structural, and each of them changes what the harness measures.

Dead code is a semantics question, not a compiler bug

Elimination of unused computation follows from the language’s observable-behaviour rules. In C and C++, an expression whose value is discarded and whose evaluation has no side effects may be removed, and a loop whose only purpose is to accumulate into a variable that is never read may be removed entirely. The translation unit contains no requirement that the work happen.

Managed runtimes have an equivalent rule with a sharper edge: code that cannot be reached and whose effects cannot be observed may be discarded even after compilation, and the just-in-time compiler re-derives this per compilation tier. A benchmark method that returns a computed value the caller ignores can be reduced to a constant return, and the call itself may then be inlined and removed.

The failure is quiet. The harness reports a very small duration, the distribution is tight because the remaining work is trivial, and the summary statistic looks like a good result rather than a broken one. Nothing in the output distinguishes a benchmark that eliminated its work from one that measured a genuinely fast operation, which is why the defence has to be built into the harness rather than applied as a post-hoc sanity check.

Making the result observable without changing the cost

The minimal fix is to consume the result in a way the compiler cannot prove useless. The canonical form is to accumulate into a value that escapes the timed region and is printed at the end.

#include <stdint.h>
#include <stdio.h>

static uint64_t hash_result = 0;

int main(void) {
    volatile uint64_t sink = 0;
    uint64_t acc = 0;
    for (long i = 0; i < 1000000; i++) {
        acc += (uint64_t)(i * 2654435761u);   /* the work under test */
    }
    sink = acc;          /* volatile store: the value must exist */
    hash_result = sink;  /* and must be observable outside the loop */
    printf("%llu\n", (unsigned long long)hash_result);
    return 0;
}

Two mechanisms are at work and they are not equivalent. A volatile store forces the value to be written and prevents the store itself from being removed, but it does not necessarily force the computation of the whole accumulator inside the loop, because the compiler can still derive a closed form for the sum and replace the loop with the arithmetic. Printing the value at the end forces the value to exist but does not force the loop to be executed with the intended iteration structure.

The reliable form for a loop benchmark is to make each iteration’s result feed the next iteration’s input, so no closed form exists and no iteration can be dropped without changing the value.

uint64_t chain(uint64_t n) {
    uint64_t state = 0x9E3779B97F4A7C15ull;
    for (uint64_t i = 0; i < n; i++) {
        state = state * 6364136223846793005ull + 1442695040888963407ull;
        state ^= state >> 33;    /* iteration i+1 needs iteration i's output */
    }
    return state;
}

This has a cost of its own: the dependency chain makes the loop latency-bound rather than throughput-bound, so the measured figure is the cost of the chain, not the throughput of the operation in isolation. That is a real trade. A benchmark of a multiply’s throughput needs independent multiplies; a benchmark of a mixed workload needs the chain. Choosing which to measure is a modelling decision, and the choice must be stated with the result.

The harness must also not measure itself

Everything the harness does per iteration lands inside the timed region if it is placed there. Allocating an input buffer per iteration, formatting a log line, or reading a clock costs more than many of the operations being compared, and it adds variance that scales with the allocation and scheduling behaviour of the host rather than with the code under test.

The structural fix is to hoist all setup out of the timed region and to time only the call. A single timed region containing $n$ calls, divided by $n$, is the usual arrangement, and the per-call overhead of the loop counter is then amortised but not removed. Unrolling reduces it further at the cost of making the harness’s own code longer and less obviously correct.

In managed runtimes the harness overhead includes the machinery that keeps results alive. The usual arrangement is a blackhole: a method that consumes a value in a way the compiler cannot analyse away, typically by storing it into a field the compiler treats as opaque or by an intrinsic the runtime provides. Writing such a sink by hand is fragile, because each runtime version may gain the analysis that defeats it. This is the reason benchmarking frameworks exist rather than hand-rolled loops: they own the sink and update it as compilers get smarter.

Constant folding across the timed region

An expression whose inputs are known at compile time is evaluated at compile time. A benchmark that calls a function with literal arguments, and whose function body is visible to the optimiser, can have the entire call replaced by a constant. The measured region then contains a store rather than a computation.

Interposition and separate compilation change this. A function defined in a shared library and called through the procedure linkage table cannot be folded at the call site, because the optimiser does not know which definition will be used at run time. That is why some harnesses put the benchmarked routine behind a function pointer or in a separate translation unit: the indirection costs a few cycles but guarantees the call happens.

The indirection has a measurable effect of its own. An indirect call through a pointer is a branch the predictor must resolve, and if the pointer is stable the predictor resolves it reliably after the first few executions; if the harness varies it, the cost rises. Either way the benchmark now measures the call plus the indirection, which is the honest cost of a dynamically dispatched operation and an overestimate of a statically known one.

Unrolling changes the thing being measured

An optimising compiler may unroll the timed loop, and a harness may unroll it deliberately. Unrolling reduces loop-control overhead per operation, which is what the benchmark wants if the operation is the subject, but it also changes register pressure and instruction scheduling. A loop that fits in registers when unrolled four times may spill when unrolled eight, and the reported cost per operation then rises with the unroll factor even though the operation is unchanged.

The measurement consequence is that the benchmark’s own shape becomes a parameter that interacts with the compiler. A result reported without the unroll factor, the dependency structure, and the compiler version is not reproducible, because a different compiler may choose a different unroll factor for the same source.

Verifying that the work survived

The check that costs least is to compare against a deliberately empty body. A harness that measures the operation and also measures a variant with the body removed gives the difference as the operation’s cost, and the difference is immune to the common overhead. If the two figures are indistinguishable, the body was removed and the measurement is void.

A second check is to inspect the generated code, either through a compiler’s assembly output or through a runtime’s compilation log. The presence of the expected instructions in the loop body, and the absence of an empty loop, is direct evidence. This is slower to automate but it catches the cases the differential test misses, such as a loop that was replaced by a closed-form computation and therefore has a body that looks plausible but executes a different number of times.

A third check is scaling: an operation whose cost is proportional to input size should show a proportional duration when the size is doubled. A body that was folded away shows a flat line, and a body that was partly hoisted shows a slope smaller than expected. Neither check proves correctness, but a failure of either proves a defect.

Where this leaves the harness

The three defences, an escaping result, a dependency chain, and a differential empty body, are not independent: each changes the cost profile of the timed region, so a benchmark built with all three measures something other than the naive version. What it measures is well defined, which is the property that matters. The alternative, a naive loop with no defence, measures an unknown mixture of the operation and the compiler’s decision about it, and cannot be compared across compiler versions at all. The question of how much of the measured time is clock overhead rather than work is the subject of Clock Sources and the Resolution Floor of a Timing Call, and the variance the harness introduces interacts with the estimation methods described in Quantiles from Finite Samples and Their Confidence Bands.