Deriving Latency Targets from Percentile Distributions

A latency objective is usually written as a percentile, and the percentile is usually the wrong representation for the decision the objective is meant to drive. The number that appears in a status page — p99 under 300 milliseconds — states that 99% of requests completed within 300 milliseconds. What the statement does not contain is any information about the other 1%, and it is that 1% that determines whether users are complaining.

The fix is not to add more percentiles. It is to invert the statement into a count form: the fraction of requests that completed within a threshold must be at least some value. This inverts an unbounded quantity into a bounded one, makes the objective aggregatable across instances, and forces the threshold to be chosen for a reason. The cost is that the tail beyond the threshold becomes invisible, so anything that matters out there needs its own threshold.

The percentile is a quantile of a distribution that cannot be stored

A percentile is a function of the full latency distribution, and the full distribution is not storable at scale. Storing every latency value for every request in a thirty-day window is prohibitively expensive for any service of consequence, so implementations approximate: fixed buckets (histograms), reservoir samples, or compact sketches. Each approximation has a shape error, and the shape error grows in the tail, which is exactly where the percentile of interest sits.

Histograms quantize. A request of 287 milliseconds landing in a bucket bounded by 250 and 500 milliseconds is indistinguishable from one at 480. The p99 computed from such a histogram is an interpolation within the bucket containing the target rank, and the interpolation assumes the values inside the bucket are uniformly distributed. That assumption is often false, particularly near cluster boundaries where a timeout or a retry mechanism piles requests up against a specific value.

The consequence for objectives is that precision in the percentile is an illusion. On a histogram with buckets at 100, 250, 500 and 1000 milliseconds, the p99 can be reported to three decimal places and still be wrong by a hundred milliseconds, because the entire resolution inside the widest bucket is zero. Prometheus documentation states this plainly: quantile estimation from histograms is approximate, and the accuracy depends on bucket layout, not on the number of samples.

Percentiles also do not compose. The p99 of a set of instances is not any function of their individual p99 values; it depends on the full distributions. If ten instances each serve 10,000 requests, and each fails its own 1% tail but in disjoint rank positions, the fleet p99 can be much worse than any instance’s p99 suggests when the offending requests are the ones that arrive in bursts to the same instance. Summaries have the same problem for a different reason: a summary computes the quantile on the instance and exports an opaque value, so the fleet-level quantile cannot be reconstructed at all. A ratio of counts, by contrast, sums exactly.

Thresholds should be chosen from the failure boundary

An event-based latency objective reads: at least 99% of valid requests complete within $T$. Two parameters are therefore in play, the threshold $T$ and the tolerable fraction. Only one of them appears in the usual statement of the objective, and the other is treated as fixed by convention.

$T$ has a natural derivation that has nothing to do with the observed distribution. It should sit at the point where a caller’s behaviour changes: the client library’s timeout, the point at which a page’s first contentful paint is blocked, the deadline propagated through a call graph, the interval at which a batch cycle must complete. A threshold chosen this way is stable across releases, which is what makes it a target rather than a measurement.

A threshold chosen at the current p99 is a measurement wearing a target’s clothes. It ratchets: as the service improves, the threshold moves down, and the service is permanently at its own frontier with no headroom. This is a common and self-defeating pattern because it guarantees the budget is consumed by ordinary variance rather than by defects.

The fraction matters as much as the threshold and is far more often copied from another team. At 99%, one request in a hundred exceeds $T$. At 99.9%, one in a thousand. The gap between these determines whether the objective is met by well-behaved code or only by redundancy. If the median latency is 50 milliseconds and $T$ is 300, a target of 99% may be satisfied by most services; a target of 99.9% in a fifteen-hop call graph requires that essentially every hop honours its own tail budget. The arithmetic of that composition is the reason distributed systems tend toward aggressive tail mitigation, as described in the tail-at-scale literature: a single slow shard in a fan-out read sets the latency of the whole response, and with enough fan-out, slow responses stop being rare.

What the tail beyond the threshold costs

Because the ratio form ignores everything past $T$, the tail needs explicit attention. The boundary case is easiest to see in the numbers: if 1% of requests exceed 300 milliseconds and the objective is 99%, the objective is exactly met, while half of the excess may be at 300.1 milliseconds and half at thirty seconds. The budget is spent either way.

The practical response is a second threshold with its own fraction, usually an order of magnitude looser in the fraction but much larger in the value, so that a thirty-second hang is a scored failure rather than an unscored one. Two thresholds on the same event are two objectives over the same denominator, and the intersection of the two policies decides the alert. Pairing them needs care: a threshold at 30 seconds scored at 99.95% is a tighter operational constraint in absolute failure count than one at 300 milliseconds scored at 99%, since 0.05% of traffic is half of the 0.1% permitted by 99%.

The compound objective also forces a decision about what a hung request is. A request that never completes produces no latency sample at all, so it is absent from both the numerator and the denominator, and the objective improves as the client gives up. Instrumenting the completion path only is the standard mistake; the correct count is of requests that were started, with a timeout treated as an explicit failure at a value above the threshold. That requirement pushes the measurement point toward the caller, for the reasons set out in the SLI ratio: numerator, denominator, and measurement point.

Queueing makes latency targets capacity targets

A latency objective at a fixed threshold is, in any system with a queue, a statement about utilization. Under the approximations of M/M/1 queueing, the mean time in system is $\frac{1}{\mu - \lambda}$ for service rate $\mu$ and arrival rate $\lambda$; as $\lambda$ approaches $\mu$, this diverges. Real systems are not M/M/1 and real latency objectives are not about the mean, but the shape survives: the response time is flat and then rises sharply as utilization approaches saturation, and the sharpness increases with the variability of the workload.

The operational consequence is that a service running comfortably at its latency objective at 40% utilization will not degrade smoothly to 60%. It may meet the objective at 40% and 55% and fail at 65%, and the transition depends on the arrival pattern as much as on the capacity. A latency objective is therefore a capacity signal, and the capacity planning question — how much headroom the objective requires — can be answered from the target itself plus an assumption about the burstiness of arrivals. A target of 99.9% under a threshold of 300 milliseconds tolerates only 0.1% of requests in the queue-delayed regime, which for a Poisson-like arrival process is a stricter utilization bound than the same threshold at 99%.

This is the mechanism behind the common observation that latency objectives are met until they are suddenly not. Nothing discontinuous happened in the system; the system crossed a region where the distribution’s tail was already the dominant term.

Instrumenting so the ratio is computable

A histogram whose buckets do not straddle the threshold cannot score the objective precisely. The bucket layout should place a boundary at or immediately below $T$, so that the fraction of requests above $T$ is computable as a sum of bucket deltas rather than by interpolation.

# buckets chosen so that the objective threshold (300ms) is a boundary
http_request_duration_seconds_bucket{le="0.1"}
http_request_duration_seconds_bucket{le="0.2"}
http_request_duration_seconds_bucket{le="0.3"}   # objective threshold
http_request_duration_seconds_bucket{le="0.5"}
http_request_duration_seconds_bucket{le="1"}
http_request_duration_seconds_bucket{le="+Inf"}

With a boundary exactly at the threshold, the fraction meeting the target is the le="0.3" bucket divided by the +Inf bucket, with no approximation at all. Summing those two counters across instances and then dividing gives the fleet ratio exactly, since counts are additive. The same layout supports investigating the tail beyond the threshold up to the next boundary, which is where the thirty-second case would be placed.

Bucket count is the cost. Each additional boundary multiplies the number of series by one, and cardinality is the dominant cost in most metrics systems. The right compromise is a handful of boundaries concentrated where decisions are made: one at the objective threshold, one at the timeout, and a few in between, with the understanding that the distribution’s shape is not being measured for its own sake.

Latency objectives are not a quality measure

The final limitation is that requests completing under the threshold are counted as good regardless of what they returned. A service that answers every request in five milliseconds with an error meets a latency objective and an availability objective separately, and a system that answers with a stale cache, a truncated payload, or a degraded recommendation satisfies both. Correctness has no natural threshold form, so it tends to be covered by a small number of boolean checks or by a separate correctness objective with a different shape, and the temptation to fold it into latency should be resisted. The general problem of indicators that do not fit the ratio form is treated in low-volume objectives and counting qualifiers.