State as a Mapping Between Addresses and Real Objects

The configuration says what should exist. The remote API holds what does exist. Neither one contains the link between them, and that link is what a state file provides. It records, for each address the configuration can refer to, the identifier of the remote object that address was bound to, the attribute values observed at the last read, and the dependency edges that were in force at the time.

Nothing about that data is derivable. An address such as a module path plus a resource type plus a name is a label invented by the configuration author. A remote identifier is a string chosen by the provider: an account-scoped ARN, a project-scoped numeric id, a generated UUID. There is no function that maps one to the other. Lose the mapping and the objects remain, but the configuration can no longer claim them, and the next plan proposes creating duplicates that the API may or may not reject.

That is why the file is treated as durable data rather than a cache. It can be rebuilt only by re-adopting each object individually, and re-adoption requires knowing which object corresponds to which address, which is exactly the information that was lost.

Three kinds of content in one document

The first kind is identity: address to remote id. This is the part that cannot be reconstructed. Terraform stores it per resource instance, with the address written in a canonical form that includes module paths and count or for_each keys, so that a resource in a module instantiated twice has two distinct addresses pointing at two distinct objects.

The second kind is the recorded attributes. These are the values returned by the API at the end of the last apply, kept so that a plan can compute a diff without a network round trip and so that values the configuration never specifies (computed fields, generated names, connection endpoints) are available to other resources. They are a cache in the strict sense: refresh replaces them, and a stale cache produces a plan that is wrong in a specific, recognizable way.

The third kind is metadata: the schema version of each resource type, the provider version that wrote it, and the serial number of the file itself. Schema version matters because providers change their internal representation over time, and an upgrade path exists to migrate old state forward. Serial numbers matter because they let the tool detect that the file changed underneath it.

A fourth kind, dependency edges, is implicit in older formats and explicit in newer ones. It exists so that a destroy can be ordered without re-reading the configuration, useful when the configuration has been deleted and only the state remains.

Why the file must be shared and locked

If two applies run at once against the same state, both read the same recorded world, both compute a diff, and both write. The second write overwrites the first. Objects created by the first apply are then absent from state while still existing remotely, which is the duplicate-resource failure in its most expensive form: the objects accrue cost and cannot be managed until someone imports them by hand.

Locking is the mechanism that prevents this, and it is why remote backends exist at all. A local file on one workstation is safe by accident, because only one process tends to touch it. A team needs a shared location with an atomic compare-and-set. Object stores provide conditional writes keyed on an entity tag or generation number; a database provides a transaction. Terraform’s S3 backend uses a lock object written conditionally, and its older DynamoDB option uses a conditional item write with a lock id. Both amount to the same primitive: write only if the value just read is still current.

The lock is advisory in the sense that it can be broken. Terraform exposes a force-unlock operation that removes the lock without checking whether the holder is alive, which is necessary after a crashed process and dangerous when the holder is merely slow. A long apply (a database cluster that takes twenty minutes to become available) holds the lock for the whole duration, so the window for a mistaken force-unlock is wide.

Serial numbers and the write that must not be lost

Every state write increments a serial. The tool reads the file, records the serial it saw, and on write asserts that the stored serial still matches. A mismatch means someone else wrote in between, and the write is rejected rather than merged. There is no merge semantics, because merging two independently computed worlds is not a well-defined operation: two applies may have created the same object twice, or deleted and recreated it, and no rule reconciles that without human judgment.

Rejection is the correct behaviour and it is also the reason a failed apply can leave the state file ahead of the world or behind it. Terraform writes state incrementally as resources complete, so a partial apply leaves a file describing exactly the resources that succeeded. The world and the file agree, and the next plan handles the rest. A crash between an API call and the state write is the dangerous case: the object exists and the file does not know. Refresh does not fix it, because refresh only reads objects the state already names. The object is orphaned until it is imported.

Refresh rewrites the cache and can change the plan

Refresh reads every object named in state and overwrites the recorded attributes. This is not a no-op: a remote change to an attribute that the configuration also sets will show up as a diff, which is the drift that detecting configuration drift is about. A remote change to a computed attribute will show up as a change in the recorded value, and downstream resources that consume it will see a changed input.

Refresh can also delete. If the API returns not-found for a recorded object, the tool removes it from state and plans to create it. That is usually right, and occasionally catastrophic: an object that was temporarily unreachable (a permission change, a wrong region in the provider configuration, a transient outage) looks identical to one that was deleted. The plan then proposes recreating it, and applying that plan destroys and rebuilds something that never went away. Providers mitigate this by distinguishing not-found from other errors, and by refusing to treat an authorization failure as absence, but the distinction depends on the API returning a clear status code, which not every API does.

Addresses are labels, and renaming is not free

Because the address is a label rather than a property of the object, changing it changes nothing remotely but breaks the mapping. Moving a resource into a module, or switching from count to for_each, rewrites addresses. Without a migration, the tool sees the old addresses as gone and the new ones as new: a plan that destroys and recreates everything.

The remedy is an explicit move, recorded either in state or in configuration. Terraform supports terraform state mv and, in recent versions, moved blocks that declare the rename so that it travels with the code and applies to every environment. The block form is strictly better for a team because it is reviewable and versioned, and because it runs during the normal plan rather than as a separate manual step that can be forgotten.

The same reasoning explains why count is fragile for anything that may be removed from the middle of a list. Index-based addresses shift when an element is deleted, so removing the second of five items renumbers the last three and turns one deletion into three replacements. for_each with a stable key avoids this: the address contains the key, not the position. The full argument belongs to count and for_each instance addressing, but the root cause is here: the state maps labels, and a label scheme that renumbers is a label scheme that reassigns identity.

What the state does not contain

State records what the provider chose to expose. Attributes marked sensitive are stored, often in plaintext, which makes the file a secret-bearing artifact and explains why encryption at rest and access control on the backend are part of the design rather than optional hardening. Attributes the API never returns cannot be verified and are recorded as they were sent, which means the file can disagree with the world and nothing will notice.

State also does not record intent. It cannot say whether a difference was deliberate. That judgment requires the configuration and a human, and it is the reason the plan is a separate artifact from the state: the state describes what happened, the plan describes what is about to, and only a reader can decide whether the second follows from the first.