Tag
autoscaling
- Jul 27, 2026 Concurrency Per Environment and the Unit a Function Platform Scales Why serverless platforms scale environments rather than requests, how per-environment concurrency changes the arithmetic, and where the queue lives when the limit is reached. cloud infrastructure
- Jul 21, 2026 Autoscaling Signals for Token Servers CPU utilization and request rate both mislead when capacity is pinned by KV cache and iteration time. Queue depth in tokens, cache occupancy, and time-to-first-token measure what matters. inference systems
- Jun 5, 2026 Provisioning Lag and the Reaction Time of a Fleet How the delay between a load increase and usable capacity determines the headroom a plan needs, why boot time and image pull dominate the loop, and how the lag compounds with forecast error. performance engineering
- Jan 7, 2026 Autoscaling as a Feedback Loop: Gain, Lag, and Why It Oscillates How a replica controller turns a metric into a desired count, why the control law is proportional rather than predictive, and how sampling lag and stabilisation windows prevent or cause oscillation. virtualization
- Apr 20, 2025 Autoscaling a Token Service with Cold Starts and Diurnal Load Why reactive scaling fails for inference replicas, how to derive the lead time a scale-out needs, and what a warm pool costs against the peaks it covers. inference systems
- Mar 5, 2025 Autoscaling Control Loops: Why the Metric Lags the Demand The feedback loop behind replica count: measurement windows, stabilization, and the delay stack that makes a fast scaler overshoot and an accurate one react too late. cloud infrastructure
- Aug 20, 2023 Autoscaling Model Servers on Queue Depth Rather Than Utilization Utilization-based autoscaling assumes work is interruptible and cheap to queue; for inference it produces oscillation, so the control signal has to be the wait time a request would experience. inference systems
- Apr 21, 2023 Autoscaling Policies and the Cost of Headroom How a scaling policy converts a load signal into an instance count, why the control loop lags the load, and what each unit of headroom costs per month. cloud infrastructure