Tag

queueing

29 articles

Oct 7, 2026 Seek Time, Rotational Latency, and Why the Average Is Not the Middle How a drive decomposes an access into seek, settle, and rotation, why the mean seek is not half the maximum, and how queueing turns latency into a throughput number. hardware engineering
Sep 17, 2026 Latency Targets and the Capacity Floor They Imply Why queueing delay grows without bound as utilization approaches one, how a p99 objective translates into a utilization ceiling, and what that ceiling costs per month. cloud infrastructure
Jul 27, 2026 Concurrency Per Environment and the Unit a Function Platform Scales Why serverless platforms scale environments rather than requests, how per-environment concurrency changes the arithmetic, and where the queue lives when the limit is reached. cloud infrastructure
Jul 25, 2026 Load Balancing Policy and Effective Capacity How assignment policy, connection reuse, and request-class knowledge change the throughput a pool can sustain, and why a policy tuned for latency can reduce total capacity. performance engineering
Jul 21, 2026 Autoscaling Signals for Token Servers CPU utilization and request rate both mislead when capacity is pinned by KV cache and iteration time. Queue depth in tokens, cache occupancy, and time-to-first-token measure what matters. inference systems
May 21, 2026 Utilization Targets and the Knee of the Latency Curve Why latency stays flat until utilization approaches one and then rises sharply, how arrival variability moves the knee, and how to locate the operating point from measurements rather than doctrine. performance engineering
Apr 11, 2026 Telling Saturation Apart From Contention A busy resource and a blocked one look identical in a latency graph. Queueing arithmetic, run-queue depth, off-CPU time, and steal time are what separate them. observability
Jan 26, 2026 Tail Drop, Buffer Bloat, and How Much Queue a Link Actually Needs A drop-tail queue signals congestion only by overflowing, and the delay it adds while full is the latency every other flow experiences. The bandwidth-delay product gives a floor, not an optimum. networking
Nov 10, 2025 Queue Time, Service Time, and What a Timeout Measures A client-side timeout covers every wait in the path, including queues the client never sees, and that decomposition explains why a server can look fast while its callers time out. backend systems
Nov 7, 2025 Sizing Regional Capacity for the Loss of a Region Why N+1 regional capacity is larger than the arithmetic suggests, how queueing turns the last few percent of utilization into latency, and where failover load comes from. cloud infrastructure
Oct 15, 2025 Headroom Arithmetic: Turning a Load Forecast into a Replica Count How arrival rate, service time, and utilization combine into a replica count, why the nonlinearity of queueing makes the last few percent of utilization decisive, and how safety margins compound. performance engineering
Aug 18, 2025 Statistical Multiplexing and the Case Against Reserved Circuits Why bursty traffic made end-to-end circuit reservation wasteful, how store-and-forward queueing turns silence into capacity, and where the packet argument stops being a win. computing history
Jun 20, 2025 Little's Law as a Consistency Check on Load Measurements How the identity connecting concurrency, arrival rate, and residence time exposes inconsistent measurement sets, plus the sampling errors and boundary conventions that make a correct system appear to violate it. performance engineering
May 18, 2025 Bounded Queues and Backpressure as Throughput Control Why an unbounded queue converts overload into latency and then into collapse, how a bound pushes the refusal upstream, and which drop policy makes refusal cheap. performance engineering
Apr 20, 2025 Autoscaling a Token Service with Cold Starts and Diurnal Load Why reactive scaling fails for inference replicas, how to derive the lead time a scale-out needs, and what a warm pool costs against the peaks it covers. inference systems
Apr 4, 2025 Queueing Bounds as a Deployment Design Test Using Little's law, utilization curves, and the Kingman approximation to decide whether a proposed deployment can meet a latency scenario before anything is built. software architecture
Jan 14, 2025 Head-of-Line Blocking in a Shared Worker Pool Why one slow request class poisons the latency of every class behind it in a FIFO queue, and which partitioning or scheduling schemes actually remove the convoy. backend systems
Nov 13, 2024 Channels, Dies, and Planes: How a Flash Controller Reaches Its Throughput A single flash die serves one operation at a time, so drive throughput comes from concurrency across channels and planes; the arithmetic of that concurrency sets the queue depth a workload needs. hardware engineering
Sep 7, 2024 Sizing a Replica Pool from Arrival Rate and Latency Target Queueing arithmetic that converts requests per second and a latency objective into a number of replicas, including the headroom that utilization hides. inference systems
Aug 16, 2024 Cycle Time, Work-in-Progress Limits, and Little's Law in Delivery Queues How the identity L = lambda W ties items in flight to delivery time, why operating near full capacity inflates waiting, and how to size a work-in-progress limit from a target cycle time. software engineering
Jul 26, 2024 Queueing Delay and the Cost of Service-Time Variance Why waiting time depends on variance rather than only on mean load, how the M/M/1 and M/G/1 results express that dependence, and what reducing variance buys. backend systems
Apr 18, 2024 Worst-Case Response Time of a CAN Message Under Fixed-Priority Arbitration Why the identifier is a priority, how the blocking and interference terms of a response-time bound are computed for a bit-stuffed frame, and where the analysis stops being optimistic. embedded systems
Nov 15, 2023 Feedback Latency as Compounding Interest A slow test suite charges every change a fixed toll and inflates batch size, which makes each failure harder to diagnose. The queueing mechanics behind that second effect. software architecture
Sep 28, 2023 Little's Law and the Size of a Request Pool Why the number of requests in flight equals arrival rate times service time, how that identity fixes a pool size, and why adding workers past that point buys no throughput. backend systems
Aug 20, 2023 Autoscaling Model Servers on Queue Depth Rather Than Utilization Utilization-based autoscaling assumes work is interruptible and cheap to queue; for inference it produces oscillation, so the control signal has to be the wait time a request would experience. inference systems
Jan 30, 2023 Decomposing Path Latency into Propagation, Serialization, and Queueing The four terms that sum to the delay across a network path, how each scales with distance, link rate, and load, and which ones a sender can actually influence. networking
Dec 26, 2022 Variance and the Effective Capacity of a Replica Pool How service-time variance and partially correlated demand shrink the throughput a pool can sustain at a latency target, and why replica count, not average load, sets the tail. performance engineering
Oct 31, 2022 Least-Outstanding Routing When Request Costs Vary Round robin assumes equal service cost and equal replica capacity. Token generation breaks both assumptions, so routers score replicas by work in flight instead of counting requests. inference systems
Dec 6, 2021 Fairness and Tail Latency in Task Schedulers Average latency hides the requests that wait behind a queue. How scheduler policy, queue discipline, and batching each move the tail, and why a fair scheduler can still produce an unfair system. language runtimes