Sizing a Replica Pool from Arrival Rate and Latency Target

A replica pool has a capacity measured in tokens per second and a demand measured in tokens per second, and the naive plan divides one by the other. That quotient is wrong in a specific and predictable direction: it produces a pool that is exactly saturated at peak, which is the operating point where queueing delay diverges. The correction is not a fudge factor applied to the result; it comes from the relationship between utilization and waiting time, and it is large enough that it changes the shape of the answer.

The service time is not the latency

The first step is separating three numbers that are routinely conflated. The service time of a request is the time a replica spends actively working on it, from admission to the last token. The wait time is the time it spends queued before any work begins. The response time is their sum, and it is what the latency target constrains. A pool sized so that every request is served immediately has service time equal to response time and is over-provisioned. A pool sized so that the aggregate service demand exactly fills the replicas has wait time approaching infinity and is under-provisioned. The useful design point lies between them, and the question is how far from saturation it must sit.

For a streaming response, the target usually applies to two separate quantities: time to first token and inter-token latency. Time to first token includes queueing plus prefill; inter-token latency is the decode step time and is largely a function of batch size rather than of queue depth. A plan that meets an inter-token target but ignores queueing can still fail badly on first-token latency, because a saturated pool queues requests for a long time before their first token appears.

Little’s law and the concurrency it implies

Little’s law states that for a stable system, the average number of items in the system equals the arrival rate times the average time an item spends in it: $L = \lambda W$. Applied to a replica pool, with $\lambda$ requests per second and $W$ seconds of mean response time, the mean number of requests in flight is $\lambda W$. That number is the concurrency the pool must be able to hold, and it must be compared against the concurrency the pool can hold, which is the number of replicas times the resident sequences each can carry.

Concurrency and arrival rate are not the same thing, and this is where capacity plans go wrong. A service handling 20 requests per second with a mean response time of 5 seconds needs 100 requests in flight. If each replica can hold 20 sequences in its batch, five replicas carry the load at zero queueing. The same 20 requests per second with a mean response time of 30 seconds needs 600 in flight, which is thirty replicas. Nothing about the arrival rate changed; the latency of the work did, and with it the memory and compute required to hold the work in progress.

This is the reason long-output workloads are expensive out of proportion to their token count. A request that streams 2,000 tokens holds a sequence slot for the entire duration, and the pool must have enough slots for every request that has arrived but not yet finished. Holding a slot costs KV cache bytes, which is the budget derived in the memory analysis; the number of slots times the bytes per slot must fit in the pool’s aggregate memory.

Why the utilization target is not 100 percent

Consider a pool of $c$ replicas each able to serve $\mu$ requests per second, so the aggregate service rate is $c\mu$, and let $\rho = \lambda / (c\mu)$ be the utilization. In the simplest queueing model, an M/M/c system, the mean wait before service is governed by the Erlang C formula, and its behavior near $\rho = 1$ is severe: the wait grows without bound as utilization approaches one, roughly in proportion to $1/(1-\rho)$.

The practical consequence is that the last few percent of utilization are the most expensive in latency. A pool at 50 percent utilization waits a fraction of a service time. At 80 percent the wait is several times larger. At 95 percent it is an order of magnitude larger again, and at 98 percent the queue is effectively unbounded for any burst. Buying that last 3 percent of utilization with a smaller pool costs far more in queueing delay than it saves in replica count.

The correct direction of the trade is to pick the utilization that meets the latency target and accept the replica count that follows. Rearranging, the number of replicas is $c \approx \lambda / (\rho_{\max} \mu)$, where $\rho_{\max}$ is the largest utilization at which the target is still met. For a service where the target is a few times the service time, $\rho_{\max}$ lands in the 0.6 to 0.8 range; for a tighter target it falls further. These are properties of the arrival process and the service time distribution, not universal constants, and they should be derived from the target rather than assumed.

Variance changes the answer, and inference traffic is variable

The M/M/c model assumes exponential service times, which have a coefficient of variation of 1. Inference service times have a much larger spread because output length varies by an order of magnitude between requests, and the sum of a random number of decode steps is itself highly variable. When the service time distribution is more variable than exponential, the queueing delay at a given utilization is larger, and the required headroom increases. The squared coefficient of variation $C_s^2$ of the service time enters the standard approximations directly, and doubling it roughly doubles the waiting contribution at fixed utilization.

Burstiness in arrivals compounds this. Inference demand arrives in correlated clusters rather than as a Poisson stream: a batch job fans out, a user interface issues several requests at once, a retry storm follows a timeout. A pool sized for the mean arrival rate with Poisson assumptions will queue heavily during correlated bursts even when its average utilization looks modest. The defensible response is to size against a peak-window arrival rate rather than a daily average, and to measure the window from the burst duration rather than from the billing period.

The calculation, with its inputs named

def replicas(arrival_rps, mean_service_s, slots_per_replica,
             target_wait_s, service_cv2=2.0):
    # Little's law: in-flight requests at zero queueing
    inflight = arrival_rps * mean_service_s
    # concurrency the pool must provide, with headroom for variance
    concurrency_needed = inflight * (1 + service_cv2 * target_wait_s
                                     / mean_service_s)
    by_concurrency = concurrency_needed / slots_per_replica

    # utilization view: solve for c with an Erlang-C style approximation
    # (illustrative; a real plan should fit the observed service distribution)
    rho_target = 0.70
    by_utilization = arrival_rps * mean_service_s / rho_target
    return max(by_concurrency, by_utilization)

# both terms must be satisfied: memory for slots, service rate for utilization

The function returns a maximum because the two constraints bind on different resources. The utilization term ensures the replicas can perform the work; the concurrency term ensures the memory exists to hold the work in progress. A pool that satisfies one and not the other fails in a characteristic way: too few replicas for the utilization shows up as queueing delay, and too few slots per replica shows up as admission rejection or preemption of in-flight sequences.

Headroom is a running cost, not a one-time decision

The number of replicas a plan produces is a floor for steady operation, and the difference between that floor and the peak requirement is capacity that sits idle most of the day. Paying for it continuously is the dominant line item in most token-serving budgets, and the alternatives are all variations on how to avoid paying for idle replicas without failing the latency target during peaks. Scale-out on queue depth, reserved instances for a known baseline, and spot capacity for the peak all trade a different cost against the same latency objective; the break-even between them is the subject of reserved and on-demand capacity.

What the queueing arithmetic contributes is a way to state the requirement honestly. A pool sized at the utilization term alone is not a lean plan; it is a plan that fails at the arrival rate it was designed for. The headroom is part of the requirement, and a plan that does not name it has not finished the calculation.