Coordinated Omission in Latency Measurement

A load generator sends a request, waits for the response, records the interval, and sends the next. The resulting latency distribution describes the intervals between request and response for requests that were actually issued. When the system slows down, the generator issues fewer requests, because it is waiting. The requests it never issued are the ones that would have experienced the slowdown, and they are absent from the distribution. The measured latency therefore improves precisely when the system is degrading, and the measured throughput falls, which makes the degradation look like a capacity change rather than a latency change.

This is coordinated omission: the measurement’s sampling is coordinated with the system’s service rate, so slow responses reduce the sample rate and remove the evidence of slowness. It is not a subtle statistical bias. It can hide delays of many seconds by the simple mechanism of not asking for them.

The mechanism, stated exactly

A closed-loop generator with one in-flight request at a time issues request i at time t_i, receives the response at t_i + d_i, and issues request i+1 at that moment. The recorded latency is d_i. The generator’s request rate is 1/d̄, so a system whose latency doubles halves the sample count.

Now suppose the correct measurement is against a schedule: requests are due at fixed intervals of 1/λ seconds regardless of when the previous one completes. If a response takes ten times the interval, the schedule accumulates nine overdue requests, each of which is delayed by the same slow response. Those nine requests’ latencies are the delays they actually experienced, which is up to ten times the recorded figure. A closed-loop generator never observes them, because it never issued them.

The bias grows with the severity of the stall. A stall of duration T against an interval of 1/λ hides approximately λT requests, each with a latency of up to T. A system that stalls for a second once a minute under a 100 request-per-second schedule omits about a hundred samples per stall from a distribution that contains six thousand samples per minute; the omitted samples are the entire tail, and the reported ninety-ninth percentile is computed from the remaining samples, which are all fast.

# Closed loop: next request is issued only after the previous completes.
def closed_loop(client, n):
    samples = []
    for _ in range(n):
        t0 = time.perf_counter()
        client.call()
        samples.append(time.perf_counter() - t0)
    return samples

# Open loop with a fixed schedule: every request is due at k/rate,
# so a slow response delays all the requests queued behind it and
# their latencies include that delay.
def open_loop(client, rate, duration):
    samples, start, k = [], time.perf_counter(), 0
    while time.perf_counter() - start < duration:
        due = start + k / rate
        while time.perf_counter() < due:
            time.sleep(max(0.0, due - time.perf_counter()))
        t0 = time.perf_counter()
        client.call()
        samples.append((time.perf_counter() - t0) + max(0.0, t0 - due))
        k += 1
    return samples

The added term in the open-loop version is the point: the sample includes the time the request was overdue before it was even issued. That is the delay the caller would have experienced had it been issuing on schedule, and it is exactly the quantity a closed-loop generator omits.

Why service-side metrics do not fix it

A server that measures the interval between receiving a request and sending the response measures service time, not residence time. The queueing that happens before the request reaches the server, in the client’s connection pool, in the load balancer, in the kernel’s accept queue, is invisible. And the queueing after the response leaves the server, on the return path, is invisible too. A borrow that waits inside the client’s pool can dominate the caller’s latency while the server’s own histogram shows a fast request, which is the split described in connection pool checkout and the exhaustion cliff.

Service-side measurement has a second, subtler version of the same bias. A server with a bounded accept queue drops connections when the queue is full; those dropped requests never appear in the service’s histogram, because they were never handled. A server that is slow enough to fill its accept queue therefore reports a distribution that excludes the requests it refused, and the refusal rate is the signal that would have revealed the problem. The histogram is a conditional distribution: latency given that the request was accepted.

Client-side measurement with an open-loop generator fixes the first problem and part of the second, but only if the generator counts its own failures and overdue requests. A generator that retries failed requests and records only successes has the same conditional-distribution problem one layer out.

The correction and what it changes

Correcting for coordinated omission means recording, for each request, the delay from when it was due to when it completed, not from when it was issued. In a closed-loop harness the correction is synthetic: when a response takes d and the intended interval is 1/λ, record d and also record the implied delays of the requests that would have been issued during the stall. Those implied samples are not measurements of anything that happened; they are a reconstruction of what an open-loop client would have experienced. Recording them changes the distribution from “what happened to the requests the generator sent” to “what would have happened to a client sending on schedule”, which is the distribution the service’s users see.

The correction’s magnitude is often the difference between meeting and missing a percentile objective. A service whose recorded ninety-ninth percentile is 50 milliseconds can have a corrected ninety-ninth percentile in the seconds, because the correction fills in the stalls. The uncorrected figure is not wrong about the requests it measured; it is answering a different question.

A second correction concerns the histogram’s resolution. Percentiles near the tail are computed from few samples, so a histogram with coarse buckets can report a ninety-ninth percentile that is really the bucket boundary rather than the observed value. High-dynamic-range histograms with logarithmic buckets and a bounded relative error address this, at a memory cost that grows with the range and the precision. The alternative — storing every sample — is exact but does not scale to high request rates.

Coordinated omission in the other direction

The bias has a mirror image: a measurement scheme that includes delay the user does not experience. A service that processes requests in parallel and records, for each request, the time until the last response in its batch has the same maximum-of-many effect that fan-out produces, and it reports a latency the caller never waited for. That is the opposite error, and it is common in systems that measure at the batch boundary rather than at the request boundary.

The general rule is that the measurement boundary must match the experience boundary. If the caller waits for one response, measure one response. If the caller waits for a page assembled from twenty calls, measure the page. Mixing boundaries produces a number that is neither the component’s latency nor the request’s, and comparing such numbers across systems is meaningless. The fan-out arithmetic in fan-out and the amplification of tail latency is the same statement from the design side: the distribution that matters is the one at the boundary the caller observes.

Percentiles over time windows

A single percentile over a long window has its own problem. A ninety-ninth percentile computed over an hour is dominated by the hour’s typical behaviour; a two-minute stall affects a small fraction of the hour’s samples and may not move the percentile at all. The stall is visible in a per-minute percentile series and invisible in the hourly aggregate.

The practical resolution is to compute percentiles over short windows and report the series, or to report the maximum over windows of a percentile. The second is the more useful for objectives, because it answers the question a user actually asks: whether there was any minute in which the service was bad. A service that meets its percentile in every minute meets it overall; a service that meets it overall can still have minutes in which nearly every request was slow.

Two further reporting choices matter. Reporting the mean alongside percentiles is misleading because the mean is dominated by the fast majority and moves little when the tail moves. Reporting a percentile without the sample count is uninterpretable, since a ninety-ninth percentile over fifty samples is the maximum of fifty samples and carries all the variance of a maximum. Both are cheap to fix and both are routinely omitted.

What a latency objective should specify

An objective that says “ninety-ninth percentile below 200 milliseconds” leaves three things open: which requests count, over what window, and whether the measurement is corrected. The first is the boundary question above. The second is the window question. The third determines whether the figure is comparable with another system’s figure at all.

The defensible form names all three: the fraction of requests, measured at the caller’s boundary, including requests that were refused or timed out, over a window of a stated length, with overdue requests accounted for. Such an objective is harder to meet than the loose form, and that is the point: the loose form is easy to meet because it excludes the requests that were hardest to serve.

There is a limit to what any single number can express. A distribution with a fast majority and a slow minority is better described by two numbers — the fraction below a threshold and the latency of the rest — than by a percentile that falls in the gap between the modes. A service whose ninety-ninth percentile is 50 milliseconds and whose ninety-ninth-point-nine is 5 seconds has a bimodal distribution, and no single percentile communicates both modes. Reporting the fraction of requests above each of several thresholds, a scheme sometimes called a latency SLI histogram, does.

The measurement is not an afterthought to the performance work; it determines which of the mechanisms in this cluster are visible at all. A pool sized by Little’s law, a fan-out bounded by amplification arithmetic, and a batch size chosen for amortization are all decisions whose consequences appear in the tail, and a measurement that omits the tail reports success for every one of them.