Modeling Throughput Growth and Service Demand Drift

A capacity plan that forecasts requests and multiplies by a fixed cost per request has a hidden assumption: that the resource needed per unit of work is constant. In practice it changes for three independent reasons, and the three are easy to confuse. The request mix shifts toward expensive endpoints. The same endpoint becomes cheaper as code is optimized. The same endpoint becomes more expensive as dependencies slow down, caches miss more, or features add work to a hot path. A forecast that treats the product as a single growth series cannot tell which is happening, and cannot say whether the answer is more hardware or less work per request.

Decomposing the capacity series

The quantity the plan actually needs is resource demand, not request count. For a single resource, demand per interval is the sum over request classes of the class’s request count times the class’s mean service demand on that resource. Writing $D$ for total demand, $N_c$ for the count of class $c$ and $s_c$ for its demand,

$$D = \sum_c N_c , s_c$$

Log differentiation gives the growth rate of demand as a weighted sum of the growth rates of the class counts plus the growth rates of the class demands, with weights equal to each class’s share of demand:

$$\frac{\dot D}{D} = \sum_c w_c \frac{\dot N_c}{N_c} + \sum_c w_c \frac{\dot s_c}{s_c}$$

The first sum is traffic growth. The second is everything else, and it is the term most plans omit. When it is small and stable, capacity grows with traffic and a simple forecast works. When it is not, the error compounds over the planning horizon: a per-request cost rising by one percent per week doubles demand in about seventy weeks, which is within the lifetime of a typical hardware purchase.

Separating the two requires per-class instrumentation. Aggregate CPU utilization against total request rate gives one ratio that mixes all classes and all releases, and that ratio will move whenever the mix moves even if no class changed. Per-endpoint demand, weighted by that endpoint’s share of requests, gives a decomposition in which a mix shift appears as a change in weights rather than a change in cost.

Why demand drifts upward

The upward drift has a small number of recurring causes, and they are worth listing because each has a different remedy.

Feature accretion is the most common. A handler gains validation, an extra lookup, a serialization step, or a call to a new dependency, and each addition costs a small fraction of the original. None is individually large enough to fail a review, and together they raise the per-request cost by a noticeable multiple over a few years. The mechanism is a ratchet: work is added to a path and rarely removed, since removing functionality is a harder decision than adding it.

Cache efficiency decline is the second. A cache whose key space grows with the user base has a hit rate that falls as the working set outgrows the cache, and the fall is not linear: a cache at 95 percent hit rate does five percent of the work at the origin, and one at 90 percent does twice as much, so a five-point drop in hit rate doubles origin demand. Capacity plans that treat cache hit rate as constant therefore underestimate growth exactly when the system is scaling, which is when the estimate matters.

Dependency latency drift is the third and is not drift in the service’s own code. A remote call whose latency rises from 2 ms to 6 ms makes the caller’s handlers hold resources three times as long, which raises concurrency for the same throughput, and the concurrency increase is what consumes capacity. This becomes a feedback loop when the dependency is itself under load, which is the subject of Feedback Effects That Change Service Demand.

Retry and timeout behaviour is the fourth. Raising a timeout does not change the happy path, but it converts slow requests into long-held resources, and adding a retry layer converts failures into additional load. Both show up as a higher demand per successful request without any change to the request’s own code.

Detecting drift from existing series

The decomposition suggests a regression rather than a curve fit. With per-class counts and a measure of resource consumption, a linear model over a window estimates the per-class demands directly:

import numpy as np

def estimate_demand(counts, resource):
    # counts: (n_observations, n_classes) request counts per interval
    # resource: (n_observations,) measured resource consumption per interval
    # Least squares gives demand per class in resource units per request.
    coeffs, *_ = np.linalg.lstsq(counts, resource, rcond=None)
    return coeffs

def drift_rate(counts_early, counts_late, res_early, res_late):
    # Compare per-class demand across two windows to isolate drift from mix.
    a = estimate_demand(counts_early, res_early)
    b = estimate_demand(counts_late, res_late)
    # Element-wise ratio of late to early demand, per class.
    return b / np.where(a == 0, np.nan, a)

Two properties make this usable in practice. The per-class coefficients are interpretable: a coefficient is the resource cost of one request of that class, in the same units as the resource measure. And the comparison of coefficients across two windows isolates drift from mix, because the mix is absorbed by the counts and the coefficients describe the cost at a given mix.

The estimator has failure modes worth knowing. Collinearity between classes, where two endpoints always move together, makes their individual coefficients unstable even when the total is well estimated. A class with a small share contributes little to the fit and its coefficient can be noisy or negative, so classes below a share threshold should be merged. And the model assumes demand per request is independent of load, which is exactly the assumption drift violates; when load-dependent behaviour is suspected, the regression needs an interaction term or a separate fit per load band.

Distinguishing drift from a release regression

Drift is gradual and a regression is a step, and the two need different responses. A step change in per-class demand after a deploy is a defect or an intended cost, and it is best found by comparing the fitted coefficients across the deploy boundary rather than by watching a trend line. A gradual change over months is drift, and it is best found by fitting the coefficient series with a trend.

The distinction also determines whether the capacity added in response is permanent. A step that is reverted returns the demand to its previous level, and capacity added for it becomes excess. A drift that continues will consume whatever capacity is added, so adding capacity without addressing the drift buys a fixed interval and no more. The interval is computable from the drift rate and the new headroom, which is the only honest way to state how long a capacity increase will last.

The series also carries a seasonal confound that is easy to mistake for drift. If the measurement window spans a period in which the traffic mix naturally shifts, such as a weekday-to-weekend change, the fitted coefficient for the affected classes will move without any change in cost. Fitting per day-of-week, or including a seasonal term, removes the confound; comparing two windows that cover the same days of the week is the cheapest way to get the same effect.

The feedback between load and demand

The most dangerous drift is the one that appears only at high load, because it means the capacity plan derived from measurements at low load is systematically optimistic. There are several mechanisms.

Queueing inside the service adds latency without adding CPU work, but latency is what holds connections, threads, and memory. A service with a fixed thread pool whose requests hold a thread for the duration spends more capacity per request when the duration includes waiting, so the demand on the concurrency resource rises with load even though the demand on the CPU does not.

Garbage collection and allocator behaviour also change with load. A heap that grows under retention pressure collects more often, and the collection work is charged to whatever request happens to be running, so per-request CPU cost rises as the working set grows. The effect is not present in a load test that runs for a few minutes, which is one reason short tests overstate sustainable capacity.

Cache and connection pool behaviour completes the picture. Under low load, connections are warm and caches are populated; under high load, connection establishment and cache misses happen more often, so the mean cost per request is higher precisely when the fleet is at its target utilization. The correct measurement is the cost at the target load, not at a convenient one.

The practical response is to measure demand as a function of load and to extrapolate to the target rather than from a comfortable operating point. If demand per request rises with utilization, the fixed point $\rho = \lambda S(\rho)$ can have more than one solution, and the capacity calculation must be evaluated at the higher one.

A drift budget for the sizing horizon

The output of the analysis is a number the plan can use: the rate at which per-request demand grows, with the mix held constant. Over a horizon of $H$ periods, demand grows by $(1+g)^{H}$ from drift alone, before traffic growth, where $g$ is the per-period drift. Multiplying that by the traffic forecast gives total demand, and dividing by the per-replica capacity at the target utilization gives the replica count, in the manner set out in Headroom Arithmetic: Turning a Load Forecast into a Replica Count.

The value of making the term explicit is that it becomes a budget. If the available capacity allows drift of one percent per week and the observed rate is three, the plan has two defensible responses: add capacity, or reduce the drift by finding its sources. The second is usually cheaper, and the per-class decomposition is what makes it findable, since a class whose coefficient has doubled is a specific handler to inspect rather than a general suspicion.

Finally, drift interacts with the scaling ceiling. A service whose throughput is capped by a serial fraction cannot buy its way out of drift, so a positive drift rate against a horizontal ceiling defines a deadline for an architectural change. That intersection of a growth series with a structural bound is the point at which planning stops being arithmetic and becomes a roadmap.