Little's Law and the Size of a Request Pool

A service accepts requests at a measured rate, spends a measured time on each one, and holds a fixed number of worker slots that can be occupied at once. Those three quantities, arrival rate, service time, and slots, are not independent knobs. They are tied together by an identity that holds for any system in steady state, regardless of how requests are scheduled, how long the queue is, or what distribution the service times follow. Getting the identity wrong is the most common reason a pool is sized by intuition and then behaves nothing like the intuition predicted.

The identity is Little’s law. It says that the average number of items inside a system equals the average rate at which items enter, multiplied by the average time each item spends inside. For a request pool, “inside” means accepted and not yet finished, so the count includes both the requests being served and the ones waiting for a slot. That distinction matters: the law counts occupancy, not activity, and a queue full of waiting requests is occupancy just as much as a busy worker is.

The identity holds without assumptions about the distribution

The usual derivation of queueing results assumes Poisson arrivals and exponential service times. Little’s law assumes neither. It requires only that the system is stable — that the number of arrivals over a long interval does not outrun the number of departures — and that the averages exist. A pool fed by a fixed-rate scheduler, a pool fed by bursty traffic, and a pool fed by a closed set of clients that only send the next request after the previous one returns all obey the same equation.

The proof is an accounting argument rather than a probabilistic one. Count the total time that all requests spent inside the system over an observation window, and divide by the window length. That quotient can be computed two ways: as the average number of requests present, integrated over time, or as the number of completed requests times their average residence time. Setting the two expressions equal gives the law. Nothing in the argument refers to a distribution, which is why the identity survives contact with real traffic far better than the formulas built on top of it.

There are two practical caveats. First, the averages must be taken over a window long enough to be representative; a pool measured over ten seconds during a burst reports a concurrency figure that describes the burst, not the day. Second, the law relates averages, so it constrains the mean residence time but says nothing about the tail. A pool can satisfy the identity exactly while a small fraction of requests wait a hundred times the mean, which is the subject of queueing delay and service-time variance.

Turning an arrival rate into a concurrency figure

The useful form of the law for pool sizing is the rearrangement: concurrency equals rate times time. A service that receives 500 requests per second and spends 30 milliseconds on each, including the time spent waiting for downstream calls, has 500 × 0.030 = 15 requests in flight on average. If each worker slot handles exactly one request at a time, fifteen slots are occupied on average.

Average occupancy is not a sizing answer, because occupancy fluctuates. The number of requests in flight at a given instant is a random variable whose mean is 15; on a busy second it may be 25. A pool with exactly 15 slots would therefore queue during roughly half of all seconds, and the queue would grow without bound whenever demand exceeded capacity for long enough. The standard response is to size for a utilization target rather than for the mean: divide the required concurrency by the fraction of capacity that is meant to be busy. At a target of 60 percent, 15 in-flight requests call for 25 slots.

import math

def slots_for(rate_per_second, service_seconds, target_utilization):
    """Worker slots needed to keep average utilization at or below the target."""
    in_flight = rate_per_second * service_seconds   # Little's law: L = lambda * W
    return math.ceil(in_flight / target_utilization)

# 500 req/s, 30 ms each, 60% target
slots_for(500, 0.030, 0.60)   # 25

The target utilization is not a free parameter. The wait imposed by a given utilization depends on the service-time distribution, and the curve steepens as utilization approaches one. At 60 percent the wait is modest for most distributions; at 90 percent it is several times the service time and highly sensitive to variance. The formula converts a policy decision into a number; it does not make the policy decision.

Why more workers stop adding throughput

The intuition that a bigger pool handles more traffic fails because the pool is not the bottleneck. Throughput is set by the slowest stage in the request path: the database’s commit rate, the downstream API’s capacity, the disk’s IOPS, or the CPU cost of the work itself. A worker pool larger than the concurrency the bottleneck can sustain does not increase completed requests per second. It increases the number of requests admitted into the system, and therefore the queue in front of the real constraint.

That relocation has costs that show up in the tail. Each additional in-flight request consumes a stack, a connection or a buffer, and the scheduler pays for the extra threads or tasks. When the bottleneck is a shared resource such as a lock or a connection pool, raising concurrency raises contention on that resource, and per-request service time grows as a result — the pool becomes slower per request while completing the same number per second. The effect is visible in the utilization curve described in queueing delay and service-time variance: past a point, more offered load lengthens residence time without raising departure rate.

A pool that is genuinely too small has the opposite symptom, and it is easier to diagnose. If every slot is occupied and the wait queue is non-empty while the downstream bottleneck sits well below its capacity, the pool is the constraint. If the wait queue is empty and slots are idle while throughput is capped, the pool is not the constraint and resizing it will change nothing.

Closed systems, retries, and where the identity misleads

The law applies to open systems, where arrivals come from outside and do not depend on response time. Many real clients are closed-loop: a fixed number of clients, each sending its next request only after the previous one completes. In a closed loop, slower responses reduce the arrival rate, so the system self-limits and the measured rate is an output rather than an input. Sizing from such a measurement underestimates the concurrency that an open-loop workload would produce at the same nominal client count. Load generators used for capacity work are usually closed-loop unless deliberately built otherwise, which is why their results can look optimistic once real traffic arrives.

Retries break the arithmetic in the other direction. A retried request enters the system more than once, so it contributes more than one arrival to the rate while representing a single unit of user demand. When a fraction of requests fail and are retried, the effective arrival rate at the backend is the original rate divided by one minus the retry fraction, and each retry carries its own residence time. During an incident this feedback is the mechanism by which a system that is slightly over capacity becomes dramatically over capacity: retries add load, added load raises latency, raised latency triggers timeouts, and timeouts trigger more retries. Pool sizing must therefore be done against the retry-amplified rate, not the client-visible rate.

A third distortion comes from where the clock starts. If service time is measured inside the handler but the request waited two seconds for a slot, the pool appears to need fewer slots than it does, because the wait is excluded from the average residence time. The identity still holds, but for the system as measured, which is not the system as experienced. The measurement boundary has to include queueing, as discussed in coordinated omission in latency measurement.

Choosing a number and leaving room to change it

The defensible procedure is short. Measure the arrival rate over a representative window, including the daily peak rather than the mean. Measure residence time from the moment a request is admitted to the pool to the moment the slot is released, including downstream waits. Multiply. Divide by the utilization the service-time distribution can tolerate. Compare the result with the capacity of the downstream bottleneck; if the pool figure exceeds what the bottleneck can serve, the bottleneck is the thing to fix.

Two refinements are worth keeping. First, treat the pool size as a per-instance figure and multiply by the number of instances only after checking that the bottleneck is shared — a database with a connection limit of one hundred does not care how the pool sizes are distributed across twenty application instances, and the sum is what matters. Second, revisit the figure when service time changes, because service time is the term most likely to move without notice: a new downstream call, a larger payload, or a cache that stopped hitting will shift residence time and invalidate a pool size that was correct last quarter.

The identity also explains why the same pool behaves differently at different times of day with no configuration change. Nothing about the pool changed; the arrival rate did, and the concurrency it demands moved with it. A pool sized for the peak is idle most of the day, and a pool sized for the mean queues every afternoon. The honest options are to size for the peak and accept the idle capacity, to size for a lower percentile and accept queueing during bursts, or to make the pool adaptive — and adaptation has its own failure modes, since a pool that grows in response to queue length adds concurrency exactly when the downstream bottleneck is already saturated, which is the case examined in head-of-line blocking in worker pools.