Throughput, Latency, and Why Concurrency Is Not Parallelism

A service that completes 10,000 requests per second with a mean response time of 20 milliseconds is holding roughly 200 requests at any instant. Some are executing, some are waiting for a core, some are waiting on a socket, and some are sitting in a queue that nobody instrumented. The arithmetic is $10{,}000 \times 0.020 = 200$. This is not an approximation that happens to be close; it is an identity that holds for any system in steady state, and it constrains every decision about how much work to keep in flight.

The identity is worth starting from because most throughput problems are stated in the wrong variable. An operator observes that throughput stopped rising and asks for more workers. A developer observes that latency doubled and asks for a bigger timeout. Both quantities are tied to a third one that is rarely measured, and moving any single one of the three moves the other two.

Rate, residence time, and the number in flight

Little’s law states that the average number of items in a system equals the arrival rate multiplied by the average time each item spends inside it. Written as $L = \lambda W$, it requires only that the system is stable, meaning arrivals and completions balance over the interval considered. It makes no assumption about the distribution of service times, the number of servers, or the scheduling discipline. That generality is what makes it useful as a sanity check: any measurement of the three quantities that violates the identity is wrong, and one of the three was measured over a different interval or a different boundary.

The boundary is the part that trips people up. If $L$ counts requests inside the process, then $W$ must count only the time from accept to response, not the time the connection spent in the kernel accept queue. If the pool has 64 workers and 400 requests are resident, then either the mean residence time is larger than the operators believe or the pool is not the system being measured. Choosing the boundary consistently is a prerequisite for the arithmetic to mean anything.

The same identity sizes thread pools, connection pools, and I/O queue depths. For a stage that serves each item in $S$ seconds with $c$ servers, the maximum completion rate is $c/S$, and the number of items in service at saturation is exactly $c$. A pool of 64 workers each holding a request for 5 milliseconds cannot exceed 12,800 completions per second, regardless of how fast the CPU is, because the workers are the scarce resource and the identity says so directly.

Concurrency overlaps waiting; parallelism divides work

Concurrency is the property of having several logical tasks in progress at once. Parallelism is the property of executing several of them at the same instant on different cores. A single-threaded event loop multiplexing a thousand sockets is highly concurrent and completely serial. A numerically intensive loop split across sixteen cores is parallel but has no concurrency in the task sense: there is one task, executed in pieces.

The distinction matters because the two produce throughput through different mechanisms. Concurrency raises throughput by overlapping time spent waiting. If each request spends 4 milliseconds waiting on a database and 1 millisecond consuming CPU, then eight concurrent requests can keep one core busy while the waits overlap, and the throughput ceiling becomes the database’s, not the core’s. Parallelism raises throughput by reducing the time a fixed amount of computation takes, which only helps if computation is what the time is spent on.

A pool sized for the wrong mechanism is a common defect. Sizing a CPU-bound pool to hundreds of threads adds context switches, cache pressure, and scheduler latency without adding a single useful instruction per second; the useful worker count is bounded by the number of cores. Sizing an I/O-bound pool to the core count leaves the cores idle most of the time, because the pool can only keep as many requests in flight as it has workers, and the identity then caps throughput at workers divided by service time.

The memory cost of the wrong choice is concrete. On 64-bit Linux, each thread reserves a stack whose default limit is 8 MiB of address space. The pages are faulted in lazily, so a mostly idle thread touches only a few pages, but a thousand threads still cost a thousand kernel task structures, a thousand entries in the scheduler’s run queue accounting, and a measurable amount of virtual address space. An event loop holding a thousand connections costs a descriptor and a small state structure per connection.

The narrowest stage sets the rate

Throughput of a pipeline is set by its slowest stage, not by the average. If requests pass through decoding, a database call, and serialization, and the database call saturates at 8,000 per second while the other two can do 40,000, the system does 8,000. Adding capacity to the other two stages changes nothing except the amount of memory held in queues between stages.

Finding that stage is usually a matter of measuring utilization rather than latency. A stage at 100% utilization is the constraint by definition; a stage at 30% is not, no matter how slow each individual operation looks. The subtle case is a stage that is not a separate process at all but a shared resource inside every request: a global lock, a single writer thread, an allocator arena, or a memory bus. These do not appear in a service diagram, and they saturate without any component reporting high utilization.

The serialized-resource case has a clean bound. If every item must pass through one resource that takes $t$ seconds to serve it, throughput cannot exceed $1/t$ no matter how many workers exist, because the resource serves items one at a time. A critical section of 2 microseconds caps throughput at 500,000 items per second. That number is independent of core count, which is why adding cores to a lock-bound service sometimes produces no change at all.

The knee: why full utilization is slower than partial

The identity says that if throughput is fixed and residence time grows, the number of items in flight must grow with it. Queueing theory supplies the growth rate for the simplest case, a single server with random arrivals and random service times, where mean residence time is $S/(1-\rho)$ for utilization $\rho$. The formula is idealized, but the shape it describes appears in every real queue:

Utilization	Mean residence time
0.50	$2S$
0.80	$5S$
0.90	$10S$
0.95	$20S$
0.99	$100S$

At 95% utilization, each item waits an average of nineteen service times before being served. Throughput is only 5% below the theoretical maximum, and latency is twenty times the service time. This is the trade that the phrase “latency versus throughput” refers to, and it is not a design preference: it is arithmetic that applies to any shared resource with variable arrivals.

def residence_time(service, utilization):
    # M/M/1 mean time in system, including queueing
    return service / (1.0 - utilization)

for rho in (0.5, 0.8, 0.9, 0.95, 0.99):
    print(f"{rho:5.2f}  {residence_time(0.001, rho)*1e3:8.3f} ms")

The operational consequence is that a system sized to run at 95% utilization has no capacity to absorb a burst, because the queue in front of the constrained stage is already long and any arrival above the service rate extends it further. A system at 70% has slack in the only currency that matters at that moment, which is time already spent waiting.

The other consequence is that the knee is where throughput measurements become misleading. Past it, a load increase produces almost no throughput increase and a large latency increase, and a further increase produces a collapse: queues grow without bound, memory fills, and the service stops completing work at all. The transition is not gradual, which is why the difference between 80% and 95% utilization is a difference in kind rather than degree.

Closed loops hide the bend

A load generator with a fixed number of clients is a closed system. Each client sends a request, waits for the response, and sends the next one. The offered load is therefore the client count divided by the response time, and it falls automatically when the server slows down. Such a generator cannot push a system past the knee: as the queue grows, its own offered rate shrinks to match, and the measurement settles into a stable but meaningless operating point.

An open-loop generator issues requests on a schedule independent of responses, which is what real traffic does when the clients are many and uncoordinated. Only open-loop load exposes the queueing behaviour above, and only open-loop load can show the collapse. The same distinction explains why a service that behaves well under a fixed-concurrency test can fall over in front of a traffic mix it never modelled.

What the identity implies for scaling work

Three quantities, one equation, and a choice about which one to move. Raising throughput at fixed latency requires more in-flight work, which costs memory, descriptors, or threads. Lowering latency at fixed throughput requires less in-flight work, which usually means less queueing and therefore more spare capacity. Raising both at once requires the constrained stage to get faster or to be replicated, and replication is only possible for stages that hold no shared state.

The bound on replication is the subject of Amdahl’s Law and the Serial Fraction That Caps Speedup, which turns the notion of a shared resource into a number. What happens when the queue in front of a saturated stage is deliberately bounded, rather than allowed to grow until memory runs out, is treated in Bounded Queues and Backpressure as Throughput Control.