Means, Percentiles, and What a Tail Measurement Can Answer

A latency report usually contains a mean and one or more percentiles, and the two are routinely read as if they described the same thing at different resolutions. They do not. The mean is a summary of total cost, and it is the correct input to a capacity or cost calculation. A percentile is a statement about the experience of a fraction of requests, and it is the correct input to a statement about how many users are affected. Confusing them produces decisions that are wrong in a specific, predictable direction.

What the mean is actually for

The arithmetic mean of a latency distribution equals the total time spent by all requests divided by the count. That makes it additive: the mean of two independent request streams weighted by their counts is the mean of the pooled stream, and the mean of a sum of stages is the sum of the means, provided the stages are traversed by the same requests.

Additivity is what makes the mean useful for accounting. If a service handles $10^6$ requests per second at a mean of 20 milliseconds, the aggregate time in system is $2\times 10^4$ request-seconds per second, which is the concurrency implied by Little’s Law and therefore the number of workers or connections required to sustain the rate. No percentile supports that calculation, because percentiles are not additive: the p99 of a sum of stages is not the sum of the stage p99s, and the p99 of a pool is not a weighted average of the pool members’ p99s.

The mean’s weakness is equally structural. It is dominated by whichever part of the distribution carries the most mass, which is usually the fast path, so a change confined to the slow tail barely moves it. A service where one request in a thousand takes 10 seconds instead of 20 milliseconds has a mean that rises by about 10 milliseconds, which is invisible in a report rounded to the nearest millisecond, while a thousand users out of a million experienced something the mean does not describe at all.

What a percentile claims

The p99 is the value below which 99 percent of observations fall. Two readings of that sentence are possible and only one is correct.

The correct reading is about the population: across the whole set of requests, at most one percent took longer than this value. It is a property of the sample, computed by sorting and indexing, and it says nothing about any particular request beyond the fraction it belongs to.

The incorrect reading is about an individual: that a given request has a 99 percent chance of being faster than this. That reading requires the request to be exchangeable with the sample, which fails whenever latency depends on which client, which host, or which time of day the request came from. It fails most sharply for the requests that matter, since a client that is already slow is more likely to be slow again.

The distinction becomes operational when a percentile is used to set an objective. “99 percent of requests under 200 milliseconds” is a statement about a rate of affected requests, and it is meaningful because it counts. “A typical request is under 200 milliseconds” is not implied by it and is often false: if the distribution is bimodal, with 90 percent of requests at 5 milliseconds and 10 percent at 400, the p99 is 400 and the median is 5, and neither number describes the population.

Why p99 and p99.9 are different measurements

The number of samples needed to estimate a percentile with any stability grows as the percentile moves outward. An estimate of the p99 uses the 99th-ranked sample out of 100, so with $n$ samples it is determined by a handful of order statistics; the p99.9 uses one sample in a thousand. To have even ten samples above the threshold, the window needs $10^4$ observations. At a rate of 100 requests per second, that is 100 seconds of data. At 1,000 per second it is 10 seconds. At 10 per second it is over 16 minutes.

Below that count the percentile jumps discontinuously as individual samples enter and leave the window, producing a graph that oscillates by large factors and invites both false alarms and false confidence. The oscillation is not noise in the usual sense; it is the estimate being dominated by single observations, which is a property of the estimator rather than of the system.

The practical rule is that a percentile is only reportable when the expected number of samples beyond it in the aggregation window is comfortably above a small constant, on the order of ten. A p99.9 over a one-minute window at 100 requests per second has six samples above it, which is marginal; over a five-minute window it has thirty, which is usable. The same arithmetic explains why a p99.99 is almost never measurable on a single host and why it requires fleet-level aggregation, which is exactly the problem treated in Aggregating Percentiles Across a Fleet Without Losing the Tail.

The percentile of a pool is not a pool of percentiles

A fleet of twenty hosts, each serving the same rate, has a fleet-wide p99 that is not the average of the per-host p99s. If the hosts are homogeneous, all twenty p99s are similar and the average happens to be close, which is why the error is often invisible. If one host is degraded, its p99 is far above the others, and the average of the twenty per-host p99s is pulled up by that host’s contribution at weight $1/20$, while the true fleet p99 is determined by whether the degraded host’s slow requests exceed one percent of the fleet total.

With twenty hosts at equal rate, one host’s slow requests are 5 percent of the fleet, so a degradation that affects all of that host’s traffic appears entirely in the fleet p99. The average of per-host p99s, by contrast, reflects only a twentieth of the degradation. The two numbers can differ by an order of magnitude, and the average is the one that appears on dashboards assembled from per-host summaries.

Computing the fleet percentile correctly requires the underlying counts, not the summaries. That is the argument for histograms over precomputed percentiles, since a histogram of sufficient resolution can be summed exactly and the percentile derived from the sum.

Measuring the tail without a distribution

A distribution-free way to describe a tail is the exceedance count: how many requests took longer than a threshold, per unit time. This is a counter, it is additive across hosts and across time, and it answers the question an on-call engineer actually has, which is how many users were affected and how often.

# Exceedance counting: additive, alertable, and independent of the distribution.
BUCKETS = (0.05, 0.1, 0.2, 0.5, 1.0, 2.0)   # seconds

def observe(hist, latency):
    # hist[i] = count of requests at or below BUCKETS[i], cumulative
    for i, edge in enumerate(BUCKETS):
        if latency <= edge:
            hist[i] += 1
            return
    hist[-1] += 1        # overflow: slower than the last edge

The cumulative form makes the exceedance count for any edge a single subtraction of the total, and the sum across hosts is a valid fleet-wide count because it is a sum of counts rather than of ratios. An alert phrased as “more than 500 requests per minute exceeded one second” is a statement about affected users that survives aggregation, restarts, and changes in traffic mix, none of which a percentile does.

The trade-off is resolution. A counter of exceedances at fixed thresholds cannot tell whether the slow requests took 1.1 seconds or 30, and that distinction matters for diagnosis. The resolution is a histogram with boundaries placed where the distinction matters, which is why boundary placement is a design decision rather than a default.

Reading a tail through the lens of fan-out

In a system where one user request fans out to many backend calls, the user-visible tail is not the backend tail. If a request waits for 100 independent calls and each has a one percent chance of being slow, the probability that at least one is slow is $1 - 0.99^{100} \approx 63$ percent. A backend p99 therefore becomes close to a user-visible median.

This is the quantitative core of the tail-at-scale problem, and its consequence for measurement is that a backend percentile and a frontend percentile are not comparable even when they describe the same requests. The frontend number is a statement about the maximum of many backend samples, and its distribution is the maximum’s, which has a different shape and a much heavier tail. Attributing a frontend p99 to a specific backend requires per-request tracing rather than per-service histograms, because the aggregation has already destroyed the correspondence between a slow user request and the backend calls it waited on.

The same fan-out arithmetic explains why reducing the p99 of a backend from 1 percent to 0.1 percent is worth more than reducing its median. At a fan-out of 100, the first change moves the user-visible slow-request probability from 63 percent to about 10 percent, while the second moves the median by an amount nobody can perceive.

The report that survives scrutiny

A defensible latency report states the measurement boundary, the aggregation window, the sample count, and the estimator. The boundary fixes what “a request” means; the window fixes how many samples the percentile rests on; the count makes the percentile’s stability assessable; the estimator says whether the percentile was computed from raw values or interpolated from buckets.

Without those four, a percentile is a number whose provenance is unknown, and two reports that disagree may be measuring different things rather than measuring the same thing differently. The most common instance is a comparison between a client-side p99 and a server-side p99 for the same traffic, where the difference is queueing and network time outside the server boundary, not a discrepancy in the server’s behaviour. That difference is real and worth measuring, and it is the subject of Decomposing End-to-End Latency into Waiting and Serving.