Progressive Delivery Controllers as Reconciliation Loops

An operator promoting a canary by hand performs a sequence of imperative steps: shift ten percent, wait, look at a dashboard, shift thirty percent, wait, promote. Written as a script, the sequence has a state machine in a shell variable, and any interruption (a terminal that closes, a node that restarts, a controller that crashes mid-step) leaves the system in a state the script no longer remembers. Progressive delivery tooling replaces the script with an object stored in the cluster and a controller that keeps the object’s declared state true.

That inversion is the whole design, and it is why the failure modes of automated rollouts look different from the failure modes of a script: the interesting failures are not “the step did not happen” but “two components disagree about who owns the routing object”.

Declared steps instead of executed steps

A controller in the Kubernetes sense is a loop that reads the observed state of the resources it watches and makes changes intended to bring the observed state toward the declared state. It is level-triggered: it does not need the event that caused a change, only the current state, so a controller that was offline while a resource was created will converge when it comes back. That property is what makes a rollout object survivable.

The object itself declares a strategy. In the form popularised by Argo Rollouts, a canary strategy is a list of steps, and the steps are of a small number of kinds: shift the traffic weight to a value, pause for a duration or indefinitely, run an analysis against a metrics provider and proceed only if it succeeds, or mark the new version as the stable one. Flagger’s canary object plays the same role against a different set of traffic providers. In both cases the object is declarative in the sense that matters: the controller can reconstruct where it is from the object’s status and the state of the resources it owns, without any record of what it did before.

The status field is therefore load-bearing. It records which step index the rollout has reached, the weight currently applied, and the results of any analysis runs. Reading it is how an operator answers “why is this rollout stuck”. Writing it is how the controller survives its own restart. A rollout whose progress exists only in the controller’s memory cannot be resumed after a restart, and a rollout whose progress exists in an external database reintroduces the split-brain problem the cluster object was supposed to remove.

What the controller has to own

The controller manages three things at once, and the reason to run one is that keeping them consistent by hand is the error-prone part.

First, capacity. The new version needs instances before it can serve, and the old version must keep enough instances to serve its remaining share. The naive approach of scaling the new workload up and the old one down as the weight moves is wrong, because the weight can move backward on an abort or a reversal, and capacity cannot be recalled faster than an autoscaler can provision it. The usual arrangement keeps the old version at full capacity for the whole rollout and sizes the new version for the largest weight it will carry, then retires the old one at promotion.

Second, routing. This is the part that has to be delegated, because the mechanism differs per cluster and per ingress: a Service selector swap, an annotation on an Ingress or Gateway resource consumed by the ingress controller, a service mesh virtual service with weighted destinations, or an API call to a managed load balancer. The controller is written against an interface (commonly called a traffic provider or router) that offers the operations “set the canary weight” and “promote the canary”. Adding a new ingress implementation means implementing that small interface, not rewriting the rollout logic. The interface is narrow on purpose: it can express weighted routing between two revisions of a workload, and nothing else.

Third, measurement. An analysis step names a provider and a query, evaluates it on an interval, and fails the rollout when the result crosses a threshold for a configured number of consecutive evaluations. This is where the decision rules of rollout analysis and decision rules become configuration rather than policy: the object records the interval, the failure limit, and the query, and the controller’s only job is to evaluate faithfully and stop when told.

The limits of the provider abstraction

A weight-moving interface reaches only the traffic the provider controls. Requests that do not pass through that ingress (internal service-to-service calls that resolve through cluster DNS and hit the service directly, background workers consuming from a queue, scheduled jobs, and anything arriving through a different path to the same backing store) are unaffected by the weight. A rollout controller can move the visible web traffic to a new version while a message consumer fleet continues to process with the old code, and nothing in the object expresses that. The same gap appears for anything cached upstream: a CDN that serves a cached response never reaches the router, so the fraction of user-visible requests served by the canary differs from the fraction of origin requests the weight would suggest.

The second limit is granularity. Weights are integers over a small range in most implementations, and they apply to a workload rather than to a request attribute. Any targeting that depends on the request (a specific tenant, a header, a route) must be expressed in the routing object itself, and the controller then has to know how to manipulate that object’s rule structure. This is the point at which traffic-provider implementations diverge most, because each ingress expresses rules in its own schema.

Conflicts with other controllers

The reconciliation model assumes one controller owns each object. A rollout controller that writes the weight by patching an ingress contradicts any other controller that also writes that ingress, and the two will trade writes indefinitely: each observes the other’s value as drift from its own desired state and corrects it. The symptom is a weight that oscillates between two values, or an ingress whose annotations flip in a loop, with both controllers reporting success.

The same conflict appears with GitOps reconciliation. If the routing object’s desired state is stored in a repository and synchronised continuously, then the weight the rollout controller applies is drift from the repository’s point of view, and the sync will restore the declared value — undoing the rollout step, whereupon the rollout controller re-applies it. Resolution requires that ownership be explicit: either the routing object is excluded from synchronisation and owned by the rollout controller, or the weight is declared in the repository and the rollout controller is given a field to write that the sync ignores.

Autoscaling is the other recurring conflict. A horizontal pod autoscaler that targets a workload by name does not know that during a rollout there are two workloads, and its replica-count decisions are computed from a metric that includes both versions’ contributions. Scaling the stable workload down while a canary holds a large weight reduces the capacity available for the majority of traffic; scaling the canary up changes the resource footprint of the version under evaluation, which can change the very latency signal the analysis step is reading. The usual arrangement is to autoscale a single reference (the stable workload) and let the rollout controller derive the canary’s size from it, which is a deliberate simplification: it means the canary is sized for the weight it carries, not for the load it individually observes.

What the object does not record

A rollout object records intent and progress. It does not record whether the new version is correct, whether the traffic it received was representative, or whether the analysis query measured the right thing. A controller that promotes after a successful analysis is reporting that the query did not cross its threshold, which is a statement about the query. The traffic composition problem (a canary that received a biased sample because of session affinity, or a split that never reached the code path under test) is invisible in the status field and is discussed in session affinity across versions and weighted canary routing.

The same applies to the end of the rollout. Promotion retires the old version’s capacity, and the object then records a completed state. Nothing in that state expresses whether the change can be undone; a rollout that promoted a version whose data changes are irreversible has a completion status identical to one that promoted a change of a log format. That distinction has to be maintained outside the object, which is the argument for treating irreversible steps as a separate, manually triggered change — the boundary laid out in rollback boundaries and irreversibility.