Detecting Drift Without Recreating the World

Drift is any difference between the state the configuration declares and the state the remote system is actually in. The definition is simple and the detection is not, because the tool has exactly one instrument: reading the objects it already knows about and comparing the response against what it recorded last time. Everything the instrument cannot reach is invisible, and everything it reaches is only as informative as the API’s response format.

A plan against a drifted environment shows the difference as ordinary changes. There is no separate drift report in the core workflow, and the proposed fix is usually to write the configured value back. Whether that is the right response depends on why the drift happened, which the diff does not say.

Refresh is a read, and reads are partial

The refresh phase reads each managed object by its recorded identifier. For each one, the provider calls the API, maps the response into the schema’s attributes, and replaces the recorded values. The comparison that follows is between configuration and the fresh mapping.

Three things can happen to a read. It returns an object, and the mapped attributes replace the record. It returns not-found, and the instance is removed from state so that the plan proposes creating it again. It returns an error, and the refresh fails or warns, depending on the tool and the error class.

The second case is where drift detection becomes dangerous. An object that was deleted is genuinely gone, and recreating it is correct. An object that is unreachable (because credentials lost a permission, because the provider was pointed at the wrong region, because the API returned a transient error) is indistinguishable from a deleted object unless the provider classifies the error carefully. A 403 is not a 404, and a provider that maps both to absence will propose rebuilding infrastructure that is still running and still serving traffic. Providers differ in how strictly they enforce that distinction, so an unexpected not-found in a refresh is worth treating as a question rather than an instruction.

Differences the comparison cannot see

Attribute-level comparison only covers attributes the provider models. Several classes of real change fall outside it.

Changes to objects the tool does not manage. A security group rule added by hand, a DNS record created by another process, a IAM policy attached outside the configuration: none of these have an address, so none appear in state, so none are read. The configuration is not wrong about them; it simply has no claim on them. Detecting this class requires an inventory comparison (listing everything in the account and subtracting the managed set) which is a different operation from a refresh and is what dedicated drift-detection services do. Doing it well requires knowing which unmanaged objects are legitimate, and that knowledge is not in the configuration.

Changes to nested structures that the API returns opaquely. A policy document stored as a JSON string is compared as a string unless the provider normalizes it. A server-side default inserted into a list changes the list. A field that the API reorders between calls appears as a change on every plan. These are false positives, and providers suppress them with normalizers and with schema flags that mark a field as semantically equal under a transformation. Each suppression is a place where a real change can hide.

Changes the API does not expose. Write-only fields (passwords, some keys, some tags on certain services) cannot be read back, so state holds what was sent and the comparison always agrees. A rotation performed outside the tool is invisible until something fails.

Changes inside the object’s own semantics. An object can be modified by the service itself: a managed database that fails over to another zone, a bucket whose replication status changes, an instance whose type is adjusted by an automated process. Some of these appear as attribute changes and some do not, depending on whether the provider exposes the field.

The asymmetry of remediation

Once a difference is found, there are two directions to resolve it: change the world to match the configuration, or change the configuration to match the world. The tool can only do the first. The second is a human edit followed by a plan, and choosing between them is the actual decision.

Applying the configuration back is the default and it is not neutral. If someone widened a firewall rule during an incident, the apply narrows it again, possibly during the same incident. If a service scaled a pool beyond the configured size and the configuration has a fixed size, the apply shrinks it. The plan shows these as updates and does not distinguish them from intended changes, so the reviewer has to supply the context.

The alternative (importing the change into the configuration) is how a legitimate out-of-band modification becomes permanent. That is the same operation as adopting an existing object, and it has the same requirement: the configuration must be edited to describe the object accurately before the plan will be clean. Importing existing resources covers the mechanics; the drift case adds the constraint that the import must happen before the next apply, or the apply will revert the change first.

There is a third response, which is to accept the difference and stop managing that attribute. Terraform’s lifecycle settings allow specific attributes to be ignored, so a value set by an autoscaler or a controller does not appear in the diff. This is a deliberate narrowing of the managed surface, and it is honest as long as it is written down: the attribute is no longer declared, so the tool no longer claims to control it.

Continuous reconciliation changes the question

Pull-based tools ask about drift on a schedule; push-based tools ask only when someone runs a plan. The difference is not just frequency. In a controller loop, the desired state is re-read continuously and the observed state is re-read continuously, so a divergence is corrected within one loop period, and the concept of drift collapses into “the loop has not caught up yet.”

Kubernetes makes this explicit. A controller compares spec to the observed world on every event and on a resync interval, and writes status to record what it found. There is no state file and no separate refresh, because the API server is the store and the controller’s cache is rebuilt from watches. Drift is corrected by the same mechanism that performs the initial creation: a ReplicaSet that finds three Pods when it wants five creates two, and one that finds six deletes one. Nothing distinguishes the first creation from the correction.

That model has a cost the push model avoids: the controller must be running for the world to stay converged, and it must be trusted to make changes without a human reviewing a plan. The push model’s corresponding cost is that nothing corrects drift between runs, and that the plan-time observation is stale by the time anyone reads it. Neither is strictly better; they optimize for different failure modes.

Frequency, cost, and the meaning of a clean plan

A drift check is a full refresh, so its cost is proportional to the number of managed resources and to the API’s latency and rate limits. Running it hourly on a large estate is a meaningful load on the control plane and on the provider’s rate budget, which is why drift detection is usually scheduled rather than continuous, and why some tools offer a refresh-only mode that reads without computing a change set.

A clean plan is evidence, not proof. It says the provider’s mapped view of every managed object matches the configuration. It does not say the objects are healthy, that unmanaged objects are absent, or that the service is behaving. The gap between “no changes” and “as intended” is where most operational surprises live, and closing it requires checks the provisioning tool does not perform.

One more subtlety: a clean plan can be produced by a configuration that has stopped describing anything real. If a resource was removed from the configuration, the plan proposes destroying it, which is a change. But if the resource was removed from the configuration and deleted remotely, the plan is clean, and the record of the intent is gone. The absence of a proposed change is the same whether the world matches the intent or the intent has been quietly abandoned. Version control is what distinguishes them, which is why the configuration is treated as the source of truth rather than the state.