Designing for Blast Radius in a Single Apply

A configuration is one graph and one state, and an apply operates on both as a unit. That is a convenience at small scale and a hazard at large scale, because the failure of any operation stops the scheduling of everything downstream and leaves the rest of the graph in whatever condition it reached. The measure of how much damage a single run can do is the blast radius, and it is a design property of the configuration rather than an operational accident.

Blast radius has three components. The first is the set of resources in the state being applied, since every one of them is read during refresh and potentially modified. The second is the set of resources whose replacement is implied by the proposed changes, since replacement is deletion. The third is the set that becomes unavailable during the apply even without being changed, because of ordering, because of dependency, or because the provider’s update is destructive in practice.

State boundaries are the primary lever

Two configurations with separate states fail separately. A failure in one cannot stop the other, a corrupt state in one does not block the other, and the refresh cost of each is proportional to its own contents rather than to the whole estate. This is the main argument for splitting state by lifecycle rather than by team or by service name: objects that change together belong together, and objects that change at different rates do not.

The counter-argument is coupling. Splitting a network from the workloads that use it means the workload state must reference the network’s identifiers, which are outputs of the network state. Terraform reads those through a remote state data source, which is a read at plan time of another state file. That introduces a dependency between states that the graph cannot order: the network must be applied first, by convention or by pipeline stage, and nothing in the tool enforces it.

That dependency is the price of the split, and it is usually worth paying for a boundary that matches a real failure domain. A network change that fails should not leave a fleet of application resources half-updated. Conversely, an application change that fails should not prevent a routing fix from being applied.

The number of states is a trade with its own cost. Each state has its own lock, its own backend configuration, and its own credentials path. Each cross-state reference is a read that can fail and a coupling that must be sequenced. A configuration split into fifty states has fifty places for the pipeline to be misconfigured, and the ordering between them lives in the pipeline rather than in the graph.

Replacement is deletion, and lifecycle rules are the guardrail

A plan that replaces a resource destroys the old object and creates a new one. The default order is destroy then create, which means the old object is gone before the new one exists. For a stateless web server behind a load balancer that is a capacity dip; for a database it is data loss; for a certificate it is an outage until the new one validates.

Three lifecycle settings change the behaviour. create_before_destroy reverses the order, so the new object exists before the old one is removed. It requires that the two can coexist, which means the name or identifier must differ between them; a resource with a fixed unique name cannot use it, and the failure is a conflict at create time rather than a clean error.

prevent_destroy makes any plan that would destroy the resource fail during planning, before any call. It is the strongest guardrail available because it operates at the reviewable stage. Its limitation is scope: it protects the resource it is set on, not the resources that depend on it, and it does not prevent an update that corrupts data in place. It also blocks legitimate decommissioning, which means it must be removed deliberately, and the removal is itself a change that a reviewer can see.

ignore_changes removes attributes from the comparison, which prevents a diff from being proposed at all. It is a guardrail against churn rather than against damage, and it has the side effect described in detecting and remediating drift: the ignored attribute is no longer managed, so a change to it is not drift, it is outside the model.

Ordering determines what is unavailable during the window

Within an apply, operations proceed along the graph. A resource that is being replaced is unavailable from the moment the old one is destroyed until the new one is ready, and the length of that window is the provider’s creation latency, not anything the configuration controls. A managed database that takes twenty minutes to become available produces a twenty-minute window per replacement, and the graph schedules everything downstream to wait for it.

Reducing the window means reducing the number of replacements in a single run, or ensuring that the replacements happen in an order that keeps capacity. Neither is expressible directly in the graph. The graph orders by data dependency, not by availability. A configuration can approximate the second by putting replacements in separate applies, or by splitting the state so that a rolling change is naturally staged.

Kubernetes addresses the same problem with a different vocabulary. A Deployment’s rolling update strategy declares a maximum unavailable count and a maximum surge, so the controller replaces Pods gradually and never drops below the declared capacity. A PodDisruptionBudget declares a floor for voluntary disruptions, and the eviction API refuses to remove Pods below it. Both are statements about availability during a change, made in the same object that declares the desired state. The push-based model has no equivalent primitive; availability during an apply is a property the operator arranges by sequencing applies.

What a failed apply leaves behind

Terraform writes state as operations complete. A failure therefore leaves a state that describes exactly the objects that were successfully created or modified, plus the ones that were untouched. The next plan operates on that remainder. This is a recoverable position, and it is the reason the tool does not attempt to roll back: rolling back would mean reversing operations that may themselves fail, and the resulting state would be harder to reason about than the partial one.

The unrecoverable case is a failure between an API call and the state write. The object exists and state does not know. Refresh will not find it, because refresh reads only recorded addresses. The object is orphaned until it is imported, and in the meantime it may be duplicated by the next apply, which proposes creating the resource it cannot see. This window is small and it is not zero, and the only defence is that the API rejects duplicates on a uniqueness constraint.

The other unrecoverable case is a partial destroy. An apply that deletes a parent object and then fails before deleting its children leaves children without a parent. Some APIs cascade, some refuse to delete a parent with children, and some leave the children orphaned and billable. The graph orders children before parents, so this is not supposed to happen; it happens when the graph’s edges are missing because the dependency was expressed by convention rather than by reference.

Constraints that act before the call

Several mechanisms move safety from the apply to the review.

Policy evaluation against the plan JSON is the most general. A policy can reject any plan that replaces a resource of a given type, that opens a network path, that removes a tag, or that destroys more than a stated number of resources. Because it reads the computed plan, it sees the operations rather than the configuration text, and it can be enforced in a pipeline where a human approval is not practical.

Cost estimation from the plan is a weaker but related check. It can flag a plan that adds expensive resources, though not one that removes a cheap resource that was serving traffic.

The plan’s own output is the first constraint, and it is the one most often skimmed. A plan’s summary line (the count of adds, changes, and destroys) is a cheap check against expectation. A change that was intended to update one attribute and shows a destroy is a change that will delete something, and the summary is where that becomes visible before the details are read.

None of these mechanisms constrain the provider’s in-place behaviour. An update that the API performs destructively (a resize that reboots, a policy change that briefly detaches) appears in the plan as an ordinary update. The plan describes the operation the tool requests, and the API decides what that operation means. That gap is permanent, and it is why the blast radius of a change is never fully knowable from the configuration alone.

The boundary of the design

A blast radius can be reduced to the smallest unit that still applies atomically, and no further. That unit is the state, because the state is what an apply locks and what a failure leaves partial. Everything else (the graph, the lifecycle rules, the policies) operates within that unit.

The practical consequence is that the most important design decision in a large configuration is not the module layout or the naming scheme. It is where the state boundaries are drawn, because those boundaries determine which failures are contained and which are not, and they are much harder to change after objects exist than before. Splitting a state later means moving objects between states, which is the same address-and-identity problem described in state as a mapping, performed under the constraint that both states are live and neither can be locked while the other is being written.