Tag
tail-latency
- Jun 4, 2026 Hit Ratio Arithmetic and How to Size a Cache Tier How to convert a hit ratio into source load and latency, why the miss-rate curve flattens before the working set is covered, and which measurement choices make a cache look better than it is. backend systems
- May 30, 2026 Deadline-Aware Batching and Tenant Fairness When a batch is full, admission becomes an arbitration problem. Deadline slack, tenant shares, and starvation bounds each answer a different question about who gets the next slot. inference systems
- Apr 24, 2026 Means, Percentiles, and What a Tail Measurement Can Answer Why the mean and the p99 are answers to different questions, what a percentile of a pooled sample says about an individual request, and the sub-second window where percentile estimates become unstable. performance engineering
- Apr 23, 2026 Hedged Requests, Tied Requests, and the Cost of Buying Back the Tail When issuing a second copy of a request reduces latency enough to justify the extra work, how to pick the hedge delay from the latency distribution, and why request tying costs a round trip. performance engineering
- Jan 8, 2026 Hedged Requests and the Economics of Tail Tolerance How issuing a duplicate request after a delay caps the tail, what the duplicate costs in load, and how to choose the delay so the trade stays favourable. backend systems
- Dec 27, 2025 Fan-Out and the Amplification of Tail Latency How issuing a request to many backends converts a rare slow response into a common one, the arithmetic behind that amplification, and the structural ways to cap it. backend systems
- Jul 11, 2025 Heartbeat Interval and Timeout Arithmetic for Fixed-Threshold Detectors How the choice of heartbeat period and missed-beat threshold sets false-positive rate, detection latency, and message volume at the same time, and why the tail of the delay distribution decides everything. distributed systems
- Dec 13, 2024 Work Distribution, Join Barriers, and Idle Time at the End of a Stage Why splitting a stage across parallel workers shortens the mean but lengthens the tail, what a join barrier does to a request that has one slow participant, and the arithmetic of straggler cost. performance engineering
- Sep 7, 2024 Sizing a Replica Pool from Arrival Rate and Latency Target Queueing arithmetic that converts requests per second and a latency objective into a number of replicas, including the headroom that utilization hides. inference systems
- Aug 19, 2024 Hedged Requests and the Arithmetic of Tail Latency Why the slowest of several parallel calls sets the latency of a fan-out, and how a delayed second attempt trades a fixed slice of extra load for a large reduction in the tail. backend systems
- Jul 26, 2024 Queueing Delay and the Cost of Service-Time Variance Why waiting time depends on variance rather than only on mean load, how the M/M/1 and M/G/1 results express that dependence, and what reducing variance buys. backend systems
- Jun 27, 2024 Deriving Latency Targets from Percentile Distributions Why a percentile objective is really a tail-count budget, how bucket layout limits the achievable precision, and what a latency target costs in headroom. observability
- Apr 28, 2024 Quantiles from Finite Samples and Their Confidence Bands What a p99 computed from a few hundred observations actually estimates, how the order statistic maps to a binomial interval, and why tail claims need sample sizes that frequently do not exist. performance engineering
- Dec 26, 2022 Variance and the Effective Capacity of a Replica Pool How service-time variance and partially correlated demand shrink the throughput a pool can sustain at a latency target, and why replica count, not average load, sets the tail. performance engineering
- Aug 17, 2022 Aggregating Percentiles Across a Fleet Without Losing the Tail Why per-host percentiles cannot be combined directly, how cumulative bucketed histograms make merging exact, and what temporal aggregation does to the tail that spatial aggregation does not. performance engineering
- Jul 31, 2022 Scatter-Gather Fan-Out and Tail Latency Multiplication A query over P partitions waits for the slowest one, so the 99th percentile of a single node becomes the median of the cluster. How hedged requests and partial results change the curve. distributed systems
- Dec 6, 2021 Fairness and Tail Latency in Task Schedulers Average latency hides the requests that wait behind a queue. How scheduler policy, queue discipline, and batching each move the tail, and why a fair scheduler can still produce an unfair system. language runtimes