Rolling Updates, Surge, and the Arithmetic of Availability

Replacing a set of running instances with a new version is a problem in constrained optimisation, and the constraint is not “keep the process alive”. It is “keep enough instances serving traffic that the loss of one more does not matter”. Those are different constraints, and the gap between them is where rolling updates fail: a process can be running, healthy by every internal measure, and still not serving, because it has not passed the check that the load balancer uses to decide whether to send it traffic.

The mechanism is a small state machine that maintains two replica sets and moves a single counter between them, bounded on both sides. Its correctness depends on three numbers and on one ordering guarantee.

The two bounds and the replica arithmetic

A deployment declares a replica count R and two bounds: the maximum number of pods that may exist above R during the update (maxSurge), and the maximum number that may be unavailable below R (maxUnavailable). Both are expressed either as an absolute count or as a percentage of R, and the percentage is rounded in the direction that makes the bound tighter: surge rounds up, unavailability rounds down.

From those, the controller derives two limits. The upper limit on total pods is R + maxSurge. The lower limit on available pods is R - maxUnavailable. The update proceeds by repeatedly computing how many pods to create and how many to remove such that both limits hold, given the current counts of old and new pods and the current count of available pods.

A worked case makes the interaction clear. With R = 10, maxSurge = 25% and maxUnavailable = 25%, the surge is 3 (rounded up from 2.5) and the unavailability allowance is 2 (rounded down from 2.5). The total may reach 13 and available may fall to 8. Starting from ten old pods, the controller may create three new ones, taking the total to 13. As new pods become available, old ones can be removed, but only so long as at least eight pods remain available. Because the new pods count toward availability once ready, the controller can remove an old pod for each new pod that becomes ready, keeping the total near the ceiling and availability near the floor.

The two bounds can both be zero, which is a special case. With maxSurge = 0 and maxUnavailable = 0 the update is impossible: no pod may be created above R and no pod may be removed below R, so no progress can be made. The API server does not reject this configuration, because the constraint is a relationship between two fields rather than a property of either, and the deployment simply stalls. The usual resolution is to allow a surge of one and no unavailability, which produces a strict create-then-remove sequence with full capacity throughout.

spec:
  replicas: 10
  minReadySeconds: 10
  progressDeadlineSeconds: 600
  revisionHistoryLimit: 10
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%

Readiness is the gate, not process start

The controller’s notion of “available” is not “the container is running”. A pod counts as available when it is ready and has remained ready for at least the minimum ready seconds configured on the deployment. Readiness is reported by the pod’s readiness probe, which is a periodic check (an HTTP request, a TCP connection, or an executed command) that the kubelet runs and whose success sets the pod’s ready condition.

The ordering that follows is the one that matters. A new pod is created; its containers start; the readiness probe begins running after the configured initial delay and repeats at the configured period; on success the pod becomes ready; only then does the controller count it as available and permit the removal of an old pod. If the probe is absent, a running container is treated as ready, which means the controller begins removing old pods at the moment the new process starts, before it has loaded its configuration, opened its listening socket, or warmed any cache. Traffic sent to it in that window fails.

This is the most common defect in rolling update configuration, and it is invisible in a cluster where the process happens to start fast. The symptom is a small rate of errors at every deploy, proportional to the deploy rate rather than to traffic, and it does not appear in tests that exercise a single instance.

There is a second, subtler gate: readiness must be checked continuously, not once. A pod that becomes ready and then loses readiness, because a dependency it needs has gone away, is removed from the available count and from the load balancer’s endpoint list, but the deployment controller does not roll back. It waits. If enough pods lose readiness, the update stops making progress and stays stopped, with old and new pods both present and neither set complete. The deployment has no timeout for this; the progress deadline applies to the whole rollout and causes a rollback, but only when it is configured.

The startup probe exists to separate “still starting” from “broken”. A slow-starting application with a liveness probe that fires during initialisation will be killed and restarted repeatedly, never reaching readiness. A startup probe disables the liveness and readiness checks until it succeeds, allowing a startup window that is much longer than the steady-state check interval without weakening the steady-state check.

Termination, endpoints, and the race at the other end

The removal side has a symmetric problem, and it is worse because it involves a component outside the pod. When a pod is deleted, two things happen concurrently: the pod is marked for deletion and receives SIGTERM, and the endpoint controller removes it from the set of endpoints behind a service. These are independent paths through the control plane, and the endpoint removal is not instantaneous. It is propagated to every node’s proxy, and each proxy updates its local rules.

The consequence is that a terminating pod can still receive traffic after it has begun shutting down. The standard mitigation is a pre-stop delay: the container waits, without exiting, for a configured period after SIGTERM before it stops accepting new connections and begins its shutdown. During that period the endpoint removal propagates and in-flight connections drain. The length of the delay must exceed the propagation time, which is a property of the cluster’s size and of the proxy’s sync interval, not of the application.

The termination grace period bounds the whole sequence. When it expires, the container is sent SIGKILL regardless of what it is doing. A pre-stop delay longer than the grace period is not a delay at all, because the process is killed during it, and this misconfiguration is easy to make because the two values live in different parts of the pod specification.

What the controller does when a rollout stalls

The deployment controller does not simply walk forward. It computes, on every pass, the target counts from the current state, and it also handles the case where the new replica set is not progressing.

If new pods are being created but not becoming ready, the controller keeps the old set at its current size and waits, bounded by the progress deadline if one is set. If the deadline passes, the rollout is marked failed and the controller reverses: it scales the new set down and the old set back up. The reversal is itself a rolling update with the same bounds, so a failed rollout does not restore capacity instantaneously: it walks back at the same rate it walked forward.

The revision history is what makes the reversal possible. Each change to the pod template creates a new replica set with a new revision number, and old replica sets are retained, scaled to zero, up to a configurable history limit. Retained sets are what a rollback targets, and the limit is why a rollback to a very old revision may be impossible: the set has been garbage collected. The retained sets also hold their pod templates, so they consume API server storage in proportion to the history limit and the template size, which is a real cost in a cluster with thousands of deployments.

Pausing, and the update that is not a rolling update

A deployment can be paused, which freezes the controller’s reconciliation of the replica sets while still allowing the pod template to be edited. Successive edits accumulate into a single revision that is applied when the deployment is resumed. The mechanism exists for exactly one purpose: to avoid producing a revision per edit when a template is being assembled from several sources, so that the history does not fill with intermediate states.

Pausing is not a canary. A paused deployment has no mechanism to send a fraction of traffic to the new version, and resuming it applies the whole change with the ordinary bounds. The distinct mechanism for partial exposure is a second deployment or a service-level split, which is a different design with different failure modes, covered in canary releases and progressive traffic shifting.

The rolling update’s guarantee is narrow and worth stating precisely: it maintains a floor on the number of available pods, given a readiness signal that accurately reflects serviceability, given endpoint propagation faster than the pre-stop delay, and given that the new version does not fail in a way that readiness cannot detect. The last condition is the one no controller can enforce. A version that starts, passes its probes, and returns errors for a fraction of requests is available by every measure the system has. Detecting that requires comparing outcomes between versions, which is the province of the progressive delivery mechanisms rather than of the rollout controller. The state that the rollout controller is manipulating, and the revision semantics that make rollback meaningful, are treated in controller ownership and the garbage collector.