Performance Engineering · Cluster
Latency Analysis
Queuing, tails, and coordination omission.
- 01 Little's Law: Concurrency, Arrival Rate, and Time in System Why the identity L = λW holds without any assumption about the arrival distribution, how the choice of system boundary changes all three terms at once, and where the accounting breaks when requests are dropped or retried.
- 02 Utilization and the Nonlinear Growth of Waiting Time Why queue delay rises faster than load, how the shape of the curve depends on service-time variability rather than on arrival statistics, and why a utilization target is a statement about tail risk.
- 03 Coordinated Omission and the Samples a Closed Loop Never Takes How a benchmark that waits for each response before issuing the next one systematically excludes the worst delays from its own histogram, and the interval-based correction that recovers them.
- 04 Means, Percentiles, and What a Tail Measurement Can Answer Why the mean and the p99 are answers to different questions, what a percentile of a pooled sample says about an individual request, and the sub-second window where percentile estimates become unstable.
- 05 Decomposing End-to-End Latency into Waiting and Serving How to split a request's elapsed time into the intervals where it was queued, in flight, or actually being executed, and why the sum of stage timers usually exceeds the total.
- 06 Aggregating Percentiles Across a Fleet Without Losing the Tail Why per-host percentiles cannot be combined directly, how cumulative bucketed histograms make merging exact, and what temporal aggregation does to the tail that spatial aggregation does not.
- 07 Work Distribution, Join Barriers, and Idle Time at the End of a Stage Why splitting a stage across parallel workers shortens the mean but lengthens the tail, what a join barrier does to a request that has one slow participant, and the arithmetic of straggler cost.
- 08 Bounded Queues, Backpressure, and What to Drop Under Overload Why an unbounded queue converts a capacity problem into a latency problem, how a queue bound is derived from a target latency, and the shedding policies that decide which requests are refused.
- 09 Timeout Budgets, Deadline Propagation, and the Arithmetic of Retry Amplification How a timeout that is short relative to the service latency turns into extra load, why retries must be bounded per request rather than per attempt, and how a remaining-time budget propagates across tiers.
- 10 Hedged Requests, Tied Requests, and the Cost of Buying Back the Tail When issuing a second copy of a request reduces latency enough to justify the extra work, how to pick the hedge delay from the latency distribution, and why request tying costs a round trip.