Backend Systems · Cluster
Backend Performance
Throughput, tail latency, and pools.
- 01 Little's Law and the Size of a Request Pool Why the number of requests in flight equals arrival rate times service time, how that identity fixes a pool size, and why adding workers past that point buys no throughput.
- 02 Head-of-Line Blocking in a Shared Worker Pool Why one slow request class poisons the latency of every class behind it in a FIFO queue, and which partitioning or scheduling schemes actually remove the convoy.
- 03 Fan-Out and the Amplification of Tail Latency How issuing a request to many backends converts a rare slow response into a common one, the arithmetic behind that amplification, and the structural ways to cap it.
- 04 Connection Pool Checkout and the Exhaustion Cliff What a checkout actually costs, why exhaustion degrades a service sharply rather than gradually, and how wait queues, validation, and recycling interact under load.
- 05 Queueing Delay and the Cost of Service-Time Variance Why waiting time depends on variance rather than only on mean load, how the M/M/1 and M/G/1 results express that dependence, and what reducing variance buys.
- 06 Batching and the Amortization of Per-Request Cost How grouping work into larger units trades latency for throughput, which fixed costs actually amortize, and where a batch becomes slower than the sum of its parts.
- 07 Power of Two Choices and the Cost of a Random Probe Why sampling two backends and picking the less loaded one beats both random assignment and global state, and how stale or misleading load signals degrade the guarantee.
- 08 Contention Ceilings and the Limits of Parallel Scaling Why adding workers to a shared counter, lock, or cache line stops helping, how the serial fraction bounds speedup, and which designs remove the ceiling instead of raising it.
- 09 Coordinated Omission in Latency Measurement Why measuring response time only for completed requests hides the worst delays, how the bias arises, and which measurement schemes avoid it without inventing numbers.
- 10 Hedged Requests and the Economics of Tail Tolerance How issuing a duplicate request after a delay caps the tail, what the duplicate costs in load, and how to choose the delay so the trade stays favourable.