Warmup Transients and Detecting Steady State in Timed Loops

A timed loop produces a sequence of durations, and the summary statistic of that sequence is treated as the cost of the operation. The step that is usually skipped is deciding which part of the sequence belongs to the operation at all. Early iterations run against a machine state that later iterations never see again: cold instruction cache, unmapped heap pages, an interpreter instead of compiled code, a frequency governor still ramping. Averaging across that transient reports a number that describes neither the warm nor the cold state, and the reported value moves when the iteration count changes even though the code did not.

The first iterations execute different code

In a managed runtime the difference is literal. A method starts in an interpreter or a baseline compiler, accumulates invocation and loop-back-edge counters, and is recompiled at thresholds. Each tier produces different machine code, so the same source line has several costs during a single run. The transition is not instantaneous across the whole program either: a hot caller may be optimised while a callee it invokes is still interpreted, and inlining decisions made at one tier are not revisited until a later recompilation.

Ahead-of-time compiled code has a smaller but real version of the same effect. The first pass over a function loads its instructions into the instruction cache and its data into the data cache; the second pass does not. If the working set exceeds the last-level cache, later iterations may also be evicted by the benchmark’s own output buffers, which is a different steady state than the one the first iterations saw.

Memory adds a second layer. A freshly started process faults in pages as it touches them, and the kernel zeroes each anonymous page before handing it over. An allocator that grows its arena by asking the kernel for memory pays that cost once per page, so the allocation path measured early includes page faults that the steady-state path does not. The same is true of the file cache: the first read of a data file goes to the device, later reads hit memory. A benchmark that reads its input each iteration and does not control for this is measuring the page cache after the first iteration and the device on the first.

Branch predictors and prefetchers also carry state between iterations. A loop whose branch outcome alternates in a pattern will mispredict until the predictor has learned the pattern, and a stride prefetcher needs several accesses to lock onto a stride. Both effects are small individually, on the order of tens of cycles per event, but they land disproportionately on the early iterations of a short run.

Warmup is not neutral

The intuitive model is that warmup is a prefix that can be discarded, leaving an unbiased measurement of the steady state behind. That model fails whenever the program adapts to the run.

Profile-guided optimisation is the clearest case. A runtime that compiles based on collected type feedback optimises for the inputs it has already seen. If the warmup phase uses one shape of input and the measurement phase uses another, the compiled code is tuned for the wrong case, and the measurement reports a deoptimisation or a generic path that a real workload with consistent input would never take. The reverse also happens: warmup with the measurement input produces optimistic code that does not generalise.

Operating system state adapts too. A process that has been running for a while is likely to have had its pages promoted to huge pages by the kernel’s page scanner, to have more of its input resident in the kernel’s page cache, and to have accumulated scheduler load statistics on the cores it runs on. Some of that state is what a long-lived server actually has; some of it is an artefact of the harness.

The practical consequence is that the warmup phase must use the same input distribution as the measured phase, and the reported result must be tied to the state it was measured in. A number without that qualifier is not comparable to a number from a harness with a different warmup policy.

Locating the transition instead of guessing an iteration count

Fixed warmup counts, whether ten iterations or ten thousand, encode an assumption about where the transient ends. The assumption is testable, and testing it is cheaper than defending it.

The test is to look at the series of per-iteration durations directly. A transient appears as a trend or as a change in spread, and both are visible in a windowed statistic. The coefficient of variation over a sliding window of the last $k$ iterations falls as the transient decays and then stabilises. Alternatively, the ratio of the windowed mean to the windowed median moves away from one, since a decaying transient has a right tail of slow early iterations.

import statistics

def steady_start(samples, window=30, tolerance=0.05):
    """First index where the trailing window's mean is stable to within tolerance."""
    for i in range(window, len(samples)):
        prev = samples[i - window:i]
        curr = samples[i - window + 1:i + 1]
        m_prev = statistics.fmean(prev)
        m_curr = statistics.fmean(curr)
        if m_prev == 0:
            continue
        if abs(m_curr - m_prev) / m_prev < tolerance:
            return i - window + 1
    return None

This detector answers a narrower question than “is the benchmark warm”: it finds the first window after which the mean stops moving by more than a chosen fraction. That is the property the measurement needs. It also exposes benchmarks where no such window exists, which is more informative than a fixed count that silently averages over a drifting signal.

Two details matter for correctness of the detector. The window must be long enough to average out the noise floor of the measurement itself, otherwise the criterion is satisfied by chance during a quiet stretch. And the tolerance must be compared against the run-to-run variation of the harness, not against zero: a benchmark whose iterations naturally vary by several percent cannot be shown to be stable to one percent, and a detector asked to do so will either never fire or fire on noise.

When the transient is the subject

There is a legitimate class of measurement where the cold path is the thing of interest: startup latency, first-request latency after a deploy, the cost of the first query against an uncached index. Discarding the transient destroys the result. The requirement is then to control it rather than remove it, which means starting each trial from a defined state, usually a fresh process, and reporting the full distribution including the first sample.

The two modes cannot share a harness default. A harness that always discards a warmup prefix will report the wrong number for startup work, and a harness that never does will report a blended number for steady-state work. Making the choice explicit, and recording it alongside the result, is the difference between a figure that can be compared across changes and one that cannot.

Steady state is a property of a workload, not of a program

Even a perfectly warmed process has no single steady state. Cache residency, allocator behaviour, and lock contention all depend on the request mix and concurrency level, so the steady state of a single-threaded loop over one input is not the steady state of the same function under concurrent load. The transition detector described above applies to each configuration separately, and the warmup length that suffices for one may be an order of magnitude too short for another.

This is the point where the choice of clock and the overhead of the timing call start to matter, because both change the shape of the series the detector reads. Clock Sources and the Resolution of a Timing Call covers the floor that the measurement apparatus itself imposes on the noise, and Measurement Overhead and the Code the Compiler Deletes covers the case where the timed region is optimised away entirely, which produces a perfectly stable series that measures nothing.