Where Latency Is Measured, and Why Percentiles Do Not Average

Latency is the most-reported metric in serving and the one whose meaning is least stable. Two dashboards for the same service can show p99 values that differ by an order of magnitude without either being wrong, because the clock started at different points. An aggregate computed across instances can understate the tail without any instance being misreported, because percentiles do not combine. And a number compared against a target can pass while the service is failing, because the requests that would have been slow never completed and therefore never entered the distribution.

None of this is exotic. It follows from the definitions, and the definitions are worth stating precisely because the consequences are counterintuitive.

The clock start is a design decision

A request passes through a client, a load balancer, an application server, a model server, and back. Each layer can start a timer, and the resulting distributions are nested rather than equal.

Client-observed latency includes connection establishment, TLS negotiation, queueing at the load balancer, application processing, and response transfer. Server-observed latency includes only the interval between the request being parsed and the response being written. Model-observed latency includes only the inference call. The three form a containment hierarchy, and the differences between them are the network and queueing time.

The gap is largest exactly where it matters. At the median, the layers differ by a few milliseconds. In the tail, the gap widens: a request that sits in a load balancer queue for 400 ms before reaching a server has a server-side latency that looks normal and a client-side latency that is a p99 event. Monitoring the server layer produces a dashboard that is green while users are waiting.

The practical rule is that the SLO is defined at the layer the user experiences, and every other layer’s measurement is diagnostic. A server-side histogram is how the cause is localised; it is not how compliance is judged. Systems that define the SLO on the server metric and alert on it are measuring their own processing and calling it user experience.

There is a second choice inside the server layer: whether the timer starts when the connection is accepted or when the request body is fully read. For large request bodies, such as an image upload for a vision model, the read time can dominate, and excluding it produces a latency figure that omits most of what the client waited for. There is a third: whether the timer stops when the last byte is handed to the socket or when the kernel confirms it was acknowledged. The second is what the client experiences, and the difference is the send buffer’s backlog, which grows exactly when the client or the network is slow.

A useful discipline is to record the start and end points in the metric’s definition rather than in a comment. A metric called request_duration is ambiguous in a way that costs an hour of incident analysis; one pinned to a documented pair of events can be compared across layers without ambiguity.

Percentiles do not aggregate

The p99 of a set of requests is not the average of the p99 values of the partitions. Nor is it the max, nor the median of them. It is a property of the pooled distribution and can only be computed from that distribution.

Concretely, two instances each serving 1,000 requests with p99 latencies of 100 ms and 200 ms do not produce a fleet p99 of 150 ms. The pooled p99 is the 20th slowest request out of 2,000, which depends on the shape of both distributions above their 99th percentile and is not determined by the two p99 values at all. It could be 120 ms if one instance’s tail is thin, or 190 ms if the fast instance has a few very slow outliers.

This is why aggregating percentiles across instances requires either merging the underlying histograms or accepting an approximation. A time-series system that stores a p99 per instance and averages them at query time produces a number that is not a percentile of anything.

Histogram merging is exact when the histograms share bucket boundaries and lossy otherwise. A common design stores cumulative buckets with boundaries chosen for the expected range, and merging is then a per-bucket sum followed by a quantile interpolation inside the containing bucket. The error is bounded by the bucket width, which for a well-chosen layout is a few percent. The alternative, storing a sketch per instance, gives tighter bounds at the cost of a more complex merge and a larger payload.

The same problem appears across time. A p99 over an hour is not the average of sixty per-minute p99 values. If the system stores per-minute histograms, the hourly value is computed by summing buckets across the sixty, which is correct; if it stores per-minute p99 scalars, the hourly value is unavailable.

Why the mean is not a substitute

The mean is aggregatable, which makes it tempting, and it is the wrong summary for a latency distribution because latency distributions are heavy-tailed and the mean is dominated by neither the typical nor the worst case.

A distribution with a 50 ms median, a 200 ms p99, and a small fraction of requests that take 30 seconds has a mean somewhere in the tens of milliseconds. A change that doubles the fraction of 30-second requests moves the mean noticeably while leaving the median untouched, and a change that adds 10 ms uniformly moves the median by 10 ms while barely moving the mean if the tail is large. The two numbers answer different questions and neither is a summary of the other.

The useful discipline is to report a small set of quantiles chosen for specific purposes: the median as the typical experience, a high quantile such as p99 as the tail, and a rate for “requests slower than a threshold”, because a rate is aggregatable, comparable across instances, and directly expressible as an error-budget burn.

The distribution that excludes its worst members

A latency measurement is recorded when a request completes. A request that times out, is cancelled by the client, or is killed by a deadline does not complete, and its latency is either recorded as the timeout value or not recorded at all.

Recording it as the timeout value is the better of the two, but it still distorts: every timed-out request is assigned the same latency, so a system with a 30-second timeout and 5% of requests hitting it shows a p99 of exactly 30 seconds regardless of whether those requests would have taken 31 seconds or 10 minutes. The quantile is pinned to the timeout, and improvements beyond it are invisible.

Not recording them is worse. A service that drops slow requests on timeout and computes latency over completions has a distribution that improves as the service degrades, because the degraded requests are removed from the sample. This is the mechanism behind dashboards that look healthy during an outage: the requests that would have shown the problem never returned a measurement.

The fix is to count failures separately and to make the timeout visible as its own rate. A “fraction of requests exceeding 30 seconds” metric, with timed-out requests counted in the numerator, degrades monotonically with badness and cannot be gamed by dropping slow work.

Streaming responses break the completion assumption

A generative endpoint does not have one latency. It has a time to first token, an inter-token interval distribution, and a total duration, and a user experiences all three.

Time to first token is the interval from request to the first byte of the response. It is dominated by queueing and prefill, so it is the metric that reflects admission delay. Total duration is dominated by output length, so it is a metric about the request rather than about the service. Reporting total duration as the latency figure conflates the two and makes the service look slower as users ask for longer answers.

Inter-token latency is the interval between consecutive tokens. It is what determines whether a stream feels responsive, and it is the metric most affected by batching decisions: a larger batch increases throughput and increases the per-token interval for every sequence in it.

A monitoring design that reports a single latency number for a streaming endpoint has to pick one of these and will mislead about the other two. The workable design reports time to first token as the latency SLO, total duration normalised by output length as a throughput-efficiency measure, and inter-token latency as a separate distribution. Each is computed from the same request records with different timestamps. The choice of which of these becomes the autoscaling signal is a separate decision, treated in Autoscaling on Queue Depth.

def stream_metrics(chunks):
    # chunks: (timestamp_s, token_text) pairs, in arrival order.
    first_ts, _ = chunks[0]
    last_ts, _ = chunks[-1]
    gaps = [b[0] - a[0] for a, b in zip(chunks, chunks[1:])]
    return {
        "time_to_first_token_s": first_ts - request_start,
        "total_s": last_ts - request_start,
        "inter_token_p50_s": percentile(gaps, 50),
        "token_count": len(chunks),
    }

The gaps list is one shorter than the token count by construction, which is correct: $n$ tokens have $n-1$ intervals. A system that divides total duration by token count computes an average that includes the first token’s prefill time and therefore overstates the steady-state interval.

Sampling and the tail it hides

High-volume services sample traces, and sampling is where tail metrics quietly become fiction. Uniform sampling preserves the shape of the distribution in expectation, so a p99 estimated from a 1% sample of a million requests is computed from 10,000 observations and is reasonably stable.

Tail-biased sampling, which retains all slow requests and samples fast ones, is used to keep traces of interesting cases without the volume. It is fine for debugging and wrong for metrics: the sampled population no longer has the original proportions, so any quantile computed from it is biased toward the tail.

The dangerous configuration is a system that applies tail-biased sampling for traces and then computes latency percentiles from the sampled spans rather than from the unsampled request counters. The resulting p99 is inflated and drifts with the sampling policy rather than with the service.

Separating the two paths is the fix: request-level latency counters record every request and are never sampled, while traces are sampled freely because they are used for causal analysis rather than for quantiles. The counter path is cheap, a few histogram buckets per instance, and it is the only path from which an SLO can be computed. The same separation applies to drift monitoring, where the sampled trace path and the counter path diverge in the same way; Windowed Drift Detection describes the statistical consequences of window length and multiple comparisons.

The remaining question is what the SLO target should be, and that is a question about user tolerance rather than about the system. A p99 target of 200 ms for a synchronous endpoint means one request in a hundred is slower than that, which for a service receiving thousands of requests per second is a constant stream of slow experiences. Setting the target requires deciding how many slow requests per unit time are acceptable, which is a different formulation from a percentile and often a more honest one.