Coordinated Omission and the Samples a Closed Loop Never Takes

A load generator that maintains a fixed number of clients and issues a new request only when the previous one returns produces a latency histogram with a specific and severe bias: it reports the delays of the requests it happened to send, and the requests it never sent are precisely the ones that would have been slow.

The mechanism is circular. When the server stalls, the generator’s clients are all blocked waiting, so no new requests arrive during the stall. The stall itself is charged to the few requests that were in flight, and once they complete the generator resumes at full rate. The histogram therefore contains a handful of very large samples and a great many normal ones, and the percentile computation, which counts samples, weights the many normal ones heavily. A stall that lasted ten seconds out of a sixty-second test is charged to perhaps five requests out of a hundred thousand, and the 99.9th percentile barely moves.

The arithmetic of the bias

Suppose the intended load is 1,000 requests per second, and the generator uses 10 clients so that each request is expected to take 10 milliseconds. During a one-second stall of the server, all 10 clients are blocked. They complete their requests at the end of the stall, each recording roughly one second of latency, and then resume. Ten samples of one second appear in the histogram.

If the intended rate had been maintained, 1,000 requests would have been issued during that second, and every one of them would have waited an average of about half a second, because they would have arrived uniformly across the stall. The true distribution contains 1,000 samples with a mean near 500 milliseconds; the measured distribution contains 10 samples near 1,000 milliseconds. The total delay experienced by clients is the same in both accounts, about 10 client-seconds, but the measured histogram divides that total across 10 samples instead of 1,000, so each measured sample is larger and there are a hundred times fewer of them.

The percentile effect is what matters. The 99.9th percentile of the measured histogram is drawn from the normal population, because 10 samples out of 100,000 is one hundredth of a percent, which is below the 0.1 percent threshold. The correction says the 99.9th percentile should come from the stalled population, because in the corrected account those 1,000 slow samples are a full one percent of the total. The gap between the two answers is the difference between a healthy service and one that stalls every minute.

Reconstructing the missing samples

The correction does not require an open-loop generator. It requires knowing, for each measured request, how long it should have taken according to the intended schedule, and then attributing the excess to the requests that were not issued.

The standard construction uses the intended inter-arrival interval. If the target rate is $r$ requests per second, the schedule places a request at times $0, 1/r, 2/r, \ldots$. A client that is blocked until time $t$ has missed every scheduled slot between its previous completion and $t$. Each missed slot is assigned a latency equal to the time from its scheduled start to the completion that unblocked the client, and those synthetic samples enter the histogram alongside the real ones.

def correct_omission(observations, target_interval):
    """observations: (start_time, latency) as actually measured.
    target_interval: seconds between scheduled starts, e.g. 1/rate.
    Yields corrected latency samples."""
    for start, latency in observations:
        end = start + latency
        # first scheduled slot after the previous completion
        t = start
        while t < end:
            yield end - t          # what a request arriving at t would have seen
            t += target_interval

For a 10-millisecond request under a 1-millisecond schedule, the loop yields ten samples: 10, 9, 8, down to 1 millisecond. The mean of those is 5.5 milliseconds, which is the average delay a request arriving at a uniformly random point in the interval would experience, matching the hand calculation above. For a one-second stall, the loop yields a thousand samples spread from 1,000 milliseconds down to 1 millisecond, and those thousand samples now dominate the upper percentiles as they should.

The loop’s cost is proportional to the number of slots missed, which is proportional to the product of the target rate and the stall duration. A long test at a high rate with a long stall can generate a large synthetic population; implementations typically cap the reconstruction or aggregate the missed slots into a weighted form rather than materializing each one.

The assumption the correction makes

The reconstruction assumes that requests would have arrived uniformly at the target rate during the stall. If the generator is closed-loop, this is counterfactual by construction: no real client would have sent them. The justification is that the measurement is meant to answer a question about a different system, one where clients arrive independently of the server’s state, and the correction reconstructs that system’s experience from the closed-loop data.

That assumption can be wrong in a specific way. If the real clients are themselves closed-loop, for example a fixed pool of application servers each with a connection pool of ten, then the real system does not issue requests during a stall either, and the uncorrected measurement is accurate for that system. The question of which system the numbers describe is therefore unavoidable. A latency figure from a closed-loop test describes the closed-loop system, and calling it “the service’s latency” silently substitutes one system for another.

There is a second assumption: that the target rate is the right rate. If the generator is unable to sustain the target because the server is slow, then the intended rate is a fiction, and reconstructing against it produces numbers for a load level the system never experienced. This is why the correction is usually applied with the rate that the unloaded system could sustain, or with the rate the test was designed to produce, and why the resulting percentile is reported as a statement about that rate rather than about the test as run.

Why the bias survives in practice

Three properties make the bias hard to notice.

It is invisible in the mean. The mean latency of the measured histogram and the mean of the corrected histogram are actually equal in the simple construction above, because both conserve total client-seconds divided by total requests. The bias appears only in the distribution’s shape, and specifically in the upper percentiles, so a dashboard showing average response time reports the correct average while the p99.9 is wrong by an order of magnitude.

It is self-reinforcing under increasing load. As the server slows, the generator slows with it, which reduces the offered rate, which reduces contention, which keeps the server from slowing further. The system finds an equilibrium at a load level below its true capacity, and the measurement reports stable latency at that level. The plateau is the closed-loop analogue of the knee in the utilization curve, described in Utilization and the Nonlinear Growth of Waiting Time, observed through an instrument that cannot see past it. The stable reading is the artifact; the underlying service may be capable of far more throughput with worse latency, or far less.

It is easiest to commit accidentally with concurrency-limited tooling. Any generator parameterized by “number of users” or “number of threads” is closed-loop by default. The parameters that make a generator open-loop, an arrival rate or a schedule, are often secondary options, and a test configured with a user count and a think time is closed-loop with a pace that varies with server speed.

What to report instead

The output of a corrected measurement is a distribution whose upper percentiles reflect the delays that were actually imposed on clients, including the delays imposed on clients who were not yet connected. Reporting it requires stating the intended rate and the correction method, because the same raw data yields different percentiles under different target intervals.

The complementary measurement is the one that does not need correction: a generator that issues on an absolute schedule and records, for each scheduled request, the time from its scheduled start to its completion. This is the open-loop design, and its histogram contains the slow samples directly because the generator sends them whether or not earlier requests have finished. Its cost is that it must buffer or drop requests when the server falls behind, and the choice between buffering and dropping is itself a decision about which system is being described.

A third option, and the one that avoids the argument entirely, is to measure from the client side of a real deployment rather than a synthetic one. Real clients are not uniformly open-loop either, but their arrival pattern is the one that matters, and a histogram of their observed latency needs no reconstruction because it is not conditioned on a generator’s schedule. The limitation is coverage: real traffic does not visit the load levels that a test can produce deliberately, so the two measurements answer different questions and are not substitutes.

The same conditioning problem appears in a different guise when latency is aggregated across hosts, where a per-host percentile is not a percentile of the fleet. That is treated in Aggregating Percentiles Across a Fleet Without Losing the Tail.