Latency Targets and the Capacity Floor They Imply

A service-level objective expressed in latency is a constraint on the distribution of response times, and the distribution is not a free parameter of the system. For a given service time and arrival process, the tail of the response-time distribution is determined by utilization, and it rises steeply as utilization approaches one. This means a latency objective sets an upper bound on the utilization a fleet may run at, and that bound is a lower bound on cost. Capacity cannot be reduced past the point where queueing pushes the tail above the objective, no matter how efficiently the work itself is performed.

The queue is the mechanism, and its cost is superlinear

A request arriving at a service that has no free worker waits. The wait is not a constant overhead; it is the residual life of the work already in the system, and its expected value depends on how many requests are ahead. In the simplest model, arrivals form a Poisson process at rate λ, service times are exponential with mean 1/μ, and there are c servers. Utilization is ρ = λ/(cμ), and the expected number in the system follows from the queueing formula, with the expected wait derived from it by Little’s law: the average number in the system equals arrival rate times average time in the system, L = λW.

The consequence that matters for cost is the shape of the wait as a function of ρ. For a single server, the expected waiting time is proportional to ρ/(1 − ρ) times the mean service time. At ρ = 0.5 the factor is 1; at ρ = 0.8 it is 4; at ρ = 0.9 it is 9; at ρ = 0.95 it is 19. The service time itself has not changed at any of these points. All of the growth is queueing, and it is purchased entirely by running the fleet at a higher utilization.

The growth is not a modelling artifact of the exponential assumption. The precise constant depends on the service-time distribution and the number of servers, but the divergence as ρ approaches one is general: any system in which arrivals can queue will show waiting time going to infinity as spare capacity goes to zero, because the queue is a random walk with a drift that vanishes. The practical form of the statement is that utilization is not a free variable to be maximized. It is the parameter that sets latency.

A percentile objective is a utilization ceiling

An objective stated as a percentile is more demanding than an objective stated as a mean, because the tail grows faster than the mean. The p99 of waiting time in a queueing system is several times the mean wait at moderate utilization and diverges faster as utilization rises. Translating a p99 objective into a utilization ceiling requires a model of the service-time distribution, but the ordering holds: a tighter percentile and a tighter latency bound both lower the maximum sustainable utilization.

A useful inversion is to compute, for a candidate fleet size, the utilization at which the modelled p99 equals the objective, and to compare that against the utilization the load would produce. The fleet size that satisfies the objective is the smallest one whose implied ceiling is above the actual load. The computation is the same as sizing for headroom, with the setpoint derived from the objective rather than chosen.

import math

def erlang_c(c, offered_load):
    """Probability an arrival waits, for c servers and offered load a = lambda/mu."""
    a = offered_load
    term = a**c / math.factorial(c) * c / (c - a)
    s = sum(a**k / math.factorial(k) for k in range(c))
    return term / (s + term)

def wait_p99(c, lam, mu):
    """Exact p99 of waiting time for M/M/c: P(W > t) = C(c,a) * exp(-(c*mu - lam)*t)."""
    a = lam / mu
    if a >= c:
        return float("inf")            # unstable: queue grows without bound
    pw = erlang_c(c, a)                # probability that an arrival waits at all
    if pw == 0.0:
        return 0.0
    return math.log(pw / 0.01) / (c * mu - lam)

The final expression equals the mean wait multiplied by ln(100·C(c,a))/C(c,a). For a single server at ρ = 0.9, C = 0.9 and the factor is about 5, so the p99 wait is roughly five mean waits. The factor grows as utilization rises, which is why a percentile objective costs more than a mean objective and why the gap widens as the fleet is pushed harder.

The model above assumes Poisson arrivals and exponential service times. Neither holds exactly for a real service, and the p99 it predicts should be compared against observed distributions before it is used to set a fleet size. What it does show is the structure of the decision: the required server count is a function of the arrival rate, the service time, and the percentile objective, and the marginal server buys progressively less headroom as the fleet grows. The first servers added to an undersized fleet remove most of the wait; later ones remove very little, which is why the cost curve flattens and why the objective, not the budget, should set the size.

The service time is the cheaper lever

Because the wait scales with the service time as well as with utilization, reducing service time reduces required capacity at every utilization level. The relationship is not proportional: halving the mean service time halves the offered load and therefore allows roughly half the servers at the same utilization, but it also reduces the variance contribution to the tail, which is often the larger effect at high percentiles.

This makes optimization of the work itself a capacity decision rather than a separate concern. Removing a serialized dependency, batching a set of small calls into one, or caching a repeated computation reduces service time and therefore reduces the fleet that the latency objective requires. The saving is the difference between two fleet sizes, and it is permanent, whereas the saving from running the smaller fleet at higher utilization is bounded by the objective and cannot be pushed further.

The reverse also holds: a change that adds a fixed delay to every request, such as an extra network hop or a synchronous write to a remote store, raises service time and therefore raises the capacity required to hold the same percentile. The capacity cost of the added hop is easy to overlook because the hop’s latency is small in absolute terms and the fleet increase is not obviously attributable to it.

Where the model stops applying

The queueing model assumes arrivals that are independent and a service time distribution that is stable. Real workloads violate both. Traffic arrives in bursts driven by external events, and service times depend on the request mix, which changes with load. A model fitted to a quiet period will predict a lower tail than the system delivers during a burst, and the discrepancy is largest exactly when the objective is most at risk.

A second limitation is that the model describes a single queue. A service that fans out to dependencies has a response time equal to the maximum over the dependencies plus its own work, and the maximum of several random variables has a tail that is heavier than any individual one. A fleet sized for its own queue will be undersized for the fan-out, and the shortfall appears as a tail that is worse than any component’s own percentile would suggest. This is the mechanism behind tail amplification in fan-out architectures, and it means that the capacity of a dependency is part of the capacity of the caller.

A third limitation is that the model has no notion of retries. When a request times out and is retried, the retry adds load to a system that is already at its utilization ceiling, which raises the wait, which triggers more timeouts. The loop is stable only while the retry amplification is small relative to the spare capacity, and a system running at a high utilization has almost no spare capacity, so the loop is unstable exactly where it is most likely to start. The capacity cost of preventing it is the headroom that the utilization ceiling already requires; the operational cost of ignoring it is a service that fails at a load it was nominally sized for.

The floor is a property of the objective, not of the workload

The minimum capacity is set by the objective, the service time distribution, and the arrival process. None of those are billing quantities, and none of them appear in a cost report. This is why a cost programme that works only from the invoice tends to propose reductions that the latency objective forbids, and why the proposals are rejected without a clear reason.

Making the constraint explicit requires stating the objective, measuring the distribution, and computing the ceiling. Once the ceiling exists, the cost of the objective is visible: it is the difference between the fleet at the ceiling and the fleet at a higher utilization, and it can be compared against the value of the objective. A service whose p99 objective costs a fleet twice the size of the one that would meet a p95 objective is making a choice with a price, and stating the price is the only way to make the choice deliberate.

The same arithmetic bounds what commitment instruments can achieve. A commitment sized to the floor of a fleet whose size is set by a latency objective covers capacity that is present at all hours, and the discount applies to the part that cannot be removed. A commitment sized above that floor covers capacity that only exists during peaks, which is the part most likely to change. The break-even logic in commitment instruments and break-even utilization applies directly, and the latency-derived floor is the natural quantity to commit against.