Plan and Apply Across a Fleet: Partial Convergence and the Limits of Ordering

Applying to one host is a solved problem: build the graph, check for cycles, converge. Applying to a fleet of them adds three problems that do not exist on a single node. The hosts interact, so the correct intermediate state may exist only as a property of the whole set. The plan is computed against a state that is changing while the apply runs. And the apply takes long enough that an operator will want to stop it, which creates a partially migrated fleet that must nonetheless be safe to leave running.

The plan is a snapshot of a moving target

A plan is computed by reading the current state, diffing it against the declaration, and recording the operations that would close the gap. The record is useful because it can be reviewed before execution, and because the apply can be made contingent on the plan matching what the reviewer approved.

The recording has a lifetime. Between the plan and the apply, three things can change. The declaration can change, if a commit lands. The observed state can change, if another process writes to the resource. And the relationships between resources can change, if the provider’s implementation resolves a reference at apply time rather than at plan time. The first two are detectable by re-reading before applying and comparing against the plan’s recorded inputs, which is what a plan file’s digest is for. The third is not detectable, because the reference was never in the plan; it is resolved during the apply, and it can resolve differently from the plan’s prediction.

The practical consequence is that a plan and an apply are separate operations with a check in between, and the check is only as good as the plan’s coverage of the inputs. A plan that records the desired state and the observed state covers the first two changes. A plan that omits a value read from an external system during apply, such as the next free IP address or the name of a generated resource, cannot cover the third, and the apply will produce something the reviewer did not see.

Splitting plan from apply is therefore most valuable for resources whose diffs are legible: a field changing from one value to another. It is least valuable for resources whose identity is generated, because the diff cannot show the generated name until it exists. Tools that support a plan format usually expose the same distinction in their documentation, and the honest reading of a plan is “this is what the diff looks like now”, not “this is what will happen”.

A fleet is not a bigger graph

The single-host dependency graph orders resources on one machine. Across a fleet, the edges that matter are mostly not resource-to-resource, they are host-to-host: the load balancer must not send traffic to a host while its service is restarting; the database primary must be migrated before the replicas; the queue consumer must stop before the schema changes and start after.

These edges are not in the per-host graph, and no configuration tool derives them, because the configuration of one host does not reference the leases or the health checks that create them. They have to be expressed as an ordering over batches and as a health condition between batches. The ordering is coarse: batch 1 converges fully and becomes healthy, then batch 2 starts. The health condition is the load balancer’s own opinion, which means the loop between the configuration system and the load balancer is closed through a timeout, and the timeout is the parameter that matters.

The timeout has two failure directions. Too short, and a slow-starting service is drained and re-added while still warming, which produces repeated errors without any configuration being wrong. Too long, and a genuinely failing apply stalls the rollout for the timeout duration before proceeding with a batch that will also fail. The right unit for the check is a readiness signal that the service itself reports, not elapsed time, because readiness is the property the load balancer needs; elapsed time is a proxy that holds only when starting costs a constant.

Intermediate states and the second version

The distinguishing property of a fleet apply is that the fleet spends time in a state that is not the old state and not the new one. During a rolling apply, some hosts run version A and some version B, and the interaction between them is a compatibility question that neither version’s own tests exercise.

Three shapes of incompatibility are the common ones. A schema change that adds a column with no default is fine for readers of both versions and breaks writers of the old one if the column is declared non-null without a default. A configuration key that is renamed is a problem when the consumer reads only one of the two names, so the old version on a converged host reads a key that the new declaration removed. A protocol change in the transport, such as a required header or a changed serialisation, is a problem whenever a request crosses from one version to the other, which on a fleet with a shared queue is every request.

The general technique is expand-and-contract: the change is split into two releases. The first release makes every host tolerate both the old and the new shape, and is applied to the whole fleet. The second release stops using the old shape. The intermediate state after release one is fully compatible, so the fleet can spend as long as it needs in it, and the rollout can be paused, resumed, or reversed without a compatibility question. The cost is two apply cycles for every change that is not compatible by construction, which is a real cost paid in operational attention rather than in machine time. The staging environment cannot validate this boundary unless it can reproduce a mixed-version fleet, which is a specific instance of the parity problem.

Where the change cannot be split, the batch size is the only control. A batch of one host, with the change verified before the next, bounds the blast radius to one host and bounds the exposure of the incompatible state to the interval between hosts. That bound is what makes a rollout pause meaningful: at any pause point, at most one batch has been changed.

Pausing, resuming, and the fleet as a partially applied graph

A fleet apply of any size will be interrupted. The interruption happens on purpose (a reviewer stops after a batch shows an unexpected diff), on failure (a host in a batch fails and the strategy aborts), and on the operator’s whim, which must be treated as a supported operation because the alternative is that the operator kills the process and creates an unknown state.

The design that makes interruption safe is the same one that makes single-host partial application safe: convergence, in the sense that a run that changes nothing is still a full pass over every resource. Each host is individually convergent and can be applied again from any observed state. The fleet’s safety is then a property of the per-host guarantees plus the ordering, and the ordering’s state is the batch cursor. Resuming means starting from the cursor with the same declaration; the hosts already converged report no change, and the remaining ones proceed.

What does not resume cleanly is the external state that the batch cursor does not capture: a load balancer that has drained half a batch, a lock held by a process on the previous runner, a queue whose consumers were stopped but not restarted. Each of these is a resource in some system, and the cleanest treatment is to declare them as resources in the same graph so that convergence repairs them, which brings the fleet problem back to the per-host problem. Where that is not possible, the interruption procedure has to name them, because the automated resume cannot.

What the fleet apply cannot promise

It cannot promise that the fleet is ever homogeneous during a change, and it should not be read as promising that. What it can promise is that every host is convergent, that the set of hosts at each version is known at any point, and that the transition has been tested at the boundary between versions. Those three are what make an interrupted rollout a state to be finished rather than an incident.

It also cannot substitute for the declaration being correct. A rollout applies a declaration many times, quickly, with the fleet watching; a declaration error is multiplied by the batch count and its effects appear at the boundary between versions, where they are hardest to attribute. The review that catches the error is the plan review, and its coverage is exactly the coverage of the plan, which is the reason to prefer changes whose diffs the plan can show in full.