Utilization Targets and the Knee of the Latency Curve

Every capacity target eventually becomes a number, and in practice that number is a utilization: keep the CPU below seventy percent, keep the database below sixty. The numbers are usually inherited. What matters is whether the target corresponds to a real property of the system or to a habit, because utilization is the one quantity in capacity planning that is cheap to measure, easy to misread, and directly controls how much hardware is bought.

Utilization is a ratio, and the denominator matters

The metric is the fraction of wall-clock time a resource is busy. On a single-socket machine running one process, that fraction is unambiguous. On a modern server it is not. A container limited to four cores running on a host with sixty-four measures utilization differently depending on whether the source is the cgroup CPU accounting or the node-level scheduler statistic, and the two can differ by the ratio of the limit to the machine. Read container_cpu_usage_seconds_total against the cgroup limit and utilization is well defined; read a host-level busy fraction and it reflects other tenants.

The same ambiguity appears in the service time. Utilization equals arrival rate times service demand only when both are measured at the same resource. A request that spends 15 ms on the CPU, 5 ms waiting on a database, and 30 ms streaming a response can be described by any of those numbers, and the one that predicts queueing is whichever resource is closest to saturation. Capacity work that mixes a latency-derived demand with a CPU-derived utilization computes a ratio of incompatible quantities, and the resulting target is meaningless even when both inputs were measured correctly.

A useful discipline is to tag every utilization number with three things: the resource, the scope (host, cgroup, process), and the interval over which it was averaged. Averaging is the third source of error, and it is the one that hides the problem the target exists to prevent.

Short intervals expose the peak that averages hide

A five-minute average of 65 percent on a busy server is compatible with a fifteen-second stretch at 98 percent, and latency during that stretch is what users experience. Queueing theory applies to the interval over which utilization is effectively constant, so a utilization number averaged over a window much longer than the queue’s drain time describes a system that spent most of the window idle and part of it in a regime the average does not reveal. The counter-intuitive consequence is that two services with identical five-minute utilization can have completely different latency profiles, because one has smooth demand and the other has bursts.

For sizing, this argues for measuring utilization at the resolution of the latency objective. A service with a one-second latency target is judged by utilization over seconds, not over minutes, and a service whose requests take a millisecond should be judged over even shorter windows, or by the fraction of time above a threshold rather than the mean.

There is a matching trap in the other direction. Very short sampling intervals pick up the individual requests, and a one-second utilization sample on a quiet server flips between 0 and 100 percent as single requests land. The stable quantity is the fraction of samples above a level, or the mean over an interval several multiples of the queue’s characteristic time.

Where the knee actually is

The knee of the latency curve is where waiting time begins to grow faster than throughput. For a single server with Poisson arrivals, mean response time is $S/(1-\rho)$, and its derivative with respect to $\rho$ is $S/(1-\rho)^2$. At $\rho = 0.5$ the slope is four service times per unit of utilization. At $\rho = 0.8$ it is 25. At $\rho = 0.9$ it is 100. There is no threshold at which the system fails; there is a region in which each additional percentage point of load costs an increasing multiple of its predecessor.

This means the choice of target is a statement about how much of the latency budget may be spent on queueing rather than service. If the service time is the budget, the system must run near zero utilization. If the budget allows the mean response time to be twice the service time, $W = 2S$ implies $\rho = 0.5$. For a target of three times, $\rho \approx 0.67$; for five times, $\rho = 0.8$. In the notation of mean waiting time alone, $W_q = \rho S / (1-\rho)$, a mean queue wait equal to the service time corresponds to $\rho \approx 0.618$, which is where targets in the low sixties come from when the arithmetic is done properly.

The formula is for Poisson arrivals and exponential service. Real systems violate both assumptions, and both violations push the knee earlier. The M/G/1 formula makes the dependence explicit:

$$W_q = \frac{\rho S}{1-\rho} \cdot \frac{1 + C_s^2}{2}$$

Here $C_s^2$ is the squared coefficient of variation of the service time. For exponential service, $C_s^2 = 1$ and the factor is one, reproducing the simpler formula. For deterministic service it is zero and waiting time halves. For heavy-tailed service time, $C_s^2$ can be in the tens, and the same utilization produces waiting times an order of magnitude longer. A service whose p99 service time is fifty times its median has a large $C_s^2$ even if the median looks healthy, and its knee sits well below the number the exponential formula would suggest.

Arrival variability enters separately. A Poisson process has $C_a^2 = 1$; a smooth, paced arrival stream has less; a bursty one, such as requests fanning out from a synchronous client, has more. The Kingman approximation combines the two:

$$W_q \approx \frac{\rho}{1-\rho} \cdot \frac{C_a^2 + C_s^2}{2} \cdot S$$

The factor $(C_a^2 + C_s^2)/2$ is the variability multiplier, and it is the bridge between the idealized curve and a measured one. When a measured knee falls at 55 percent rather than 70, the gap is explained by variability rather than by a mysterious scheduler defect.

Locating the knee from data

Since the exponent of the denominator is what makes the curve, a plot of mean wait against utilization on a log scale separates the two regimes clearly. The practical procedure is to gather a set of (utilization, wait) pairs from a load test or from historic operation and fit the functional form rather than to guess a target.

import math

def fit_variability(samples):
    # samples: list of (rho, mean_wait_seconds, service_time_seconds)
    # Wq ~ rho/(1-rho) * k * S  =>  k = Wq * (1-rho) / (rho*S)
    ratios = []
    for rho, wait, service in samples:
        if rho <= 0 or rho >= 1:
            continue  # outside the model's domain
        ratios.append(wait * (1 - rho) / (rho * service))
    ratios.sort()
    # Median resists the noisiest high-utilization points.
    return ratios[len(ratios) // 2]

The returned value is the variability multiplier. A value near one indicates arrivals and service close to exponential; values above two indicate that the headroom must be larger than the idealized formula suggests for a given latency objective. Because the estimate is a ratio, it survives a constant scaling error in the service time, which makes it usable even when the demand figure is approximate.

The fit also produces a diagnostic. If the ratios drift upward with utilization, the single-class model is wrong: the workload mix is changing as load rises, typically because retries or cache misses add work per request, or because a scheduling policy prioritizes short requests at low load and degrades under pressure. That drift is the signature of a system whose service demand is not independent of its load, and it invalidates the linear relationship $\rho = \lambda S$ on which the whole calculation rests. The Feedback Effects That Change Service Demand article covers what generates that drift and how to detect it.

Why the target is a range, not a number

A utilization target applied uniformly across a fleet assumes the replicas are interchangeable. They are not. Replicas on machines with different background load, different amounts of memory pressure from co-tenants, or different hardware generations reach the knee at different points, and the fleet-level utilization is dominated by the slowest. Setting the target from the fleet mean produces a subset of replicas that are effectively past the knee while the aggregate looks comfortable.

The other reason for a range is that the knee moves. Deploys change the service time distribution, dependencies change their tail behaviour, and traffic mix shifts with the day. A target chosen once from a clean measurement is a snapshot of a moving property. The measured relationship is what makes re-derivation possible, which is why recording (utilization, wait) pairs continuously is more valuable than recording the target itself.

Finally, the target has to be compared against the cost of the alternative. Headroom bought with replicas is paid for continuously; the latency it prevents is paid only during overload. The trade is not symmetric in the direction people assume: an over-provisioned service costs money every hour, while an under-provisioned one costs reputation intermittently and unpredictably. That asymmetry is why headroom exists at all, and it is why the number should be derived from a measurable curve rather than chosen for comfort.