Desired State, Observed State, and the Reconciliation Loop

A provisioning script and a provisioning configuration can produce the same infrastructure and still differ in the property that matters most: what happens on the second run. A script that creates a bucket and then writes an object into it has no opinion about whether the bucket already exists. Run it twice and it fails, or worse, succeeds differently. A configuration that declares the same bucket states a fact about the world, and the tool’s job is to make that fact true, whether that requires creating the bucket, leaving it alone, or changing it.

That shift from describing steps to describing a state is the whole of the declarative model, and everything else in this cluster follows from it: dependency graphs, plan files, state storage, drift detection. The mechanics are worth understanding because the model has sharp edges. It works only when the remote API can be read back, when the tool can tell one object from another, and when the difference between the two can be computed without side effects.

The script and the state it leaves behind

An imperative script encodes a sequence. Each step assumes the effects of the previous ones: the bucket exists, the policy is attached, the function has an ARN. The assumptions are implicit in the order, and the script’s correctness depends on the starting world matching what the author had in mind. That is why shell provisioning accumulates guard clauses (check whether the directory exists, check whether the user exists) until the script is mostly a hand-written reconciler with poor error reporting.

Guards fix the common case but not the general one. A guard asks a yes-or-no question about one attribute. It cannot answer what the object should look like as a whole, and it cannot tell a difference that matters from one that does not. The alternative is to give the tool the whole desired object and let it compute the difference. That requires three capabilities that a shell script does not need: a way to read the current object, a schema that says which fields are settable and how they compare, and an identity for the object that survives across runs.

A provider is a read-modify-write interface over a remote API

Every declarative tool of this kind sits on a plugin layer. In Terraform the plugin is called a provider; in CloudFormation the equivalent logic lives inside the resource handler for each type; in Kubernetes it is the controller for the API group. The shape is the same in all three. The plugin exposes a schema, and for each managed type it implements roughly four operations: read an object by identity, create one from desired attributes, update one in place, and delete one.

The schema is not documentation. It is the contract that makes planning possible. Each attribute carries a type, a flag for whether it is required, optional, or computed by the server, and (crucially) a flag for whether changing it requires destroying the object and creating a new one. A computed attribute is one the API assigns: an ARN, a generated hostname, a creation timestamp. The tool must record those values after creation because the configuration does not contain them, and it must not try to set them on the next run.

This is also where the model first leaks. Not every remote API is a clean object store. Some operations are asynchronous: the create call returns an identifier long before the resource is usable, so the provider must poll until a terminal state. Some attributes are write-only and cannot be read back, which means the tool cannot verify them and must trust its own record. Some APIs normalize input, storing a value in a different form than it was sent. Each of these breaks the tidy read-compare-write loop in a different way, and the provider has to paper over the break with waits, with ignore_changes-style escape hatches, or with a comparison rule that treats two representations as equal.

The same contract appears in a different vocabulary in Kubernetes. Objects have a spec, which is the desired state written by the client, and a status, which is written by the controller. The API server stores both, and the controller’s loop reads the spec, compares it against what it observes in the world, and writes status and takes action. Splitting desired from observed inside the object itself is a design choice with a cost: it doubles the schema, and it means a client can never be quite sure whether a field it set has been acted on yet.

Convergence, not execution

Given a desired object and an observed object, the tool computes a diff and applies the operations the diff implies. Run it again with no external changes and the diff is empty, so nothing happens. That property (idempotence at the level of the whole configuration) is what makes repeated application safe, and it is what allows the same configuration to be applied to a fresh account and to an account with a decade of history.

Ordering is not in the configuration. When one resource refers to another’s attribute, the reference creates an edge in a dependency graph, and the tool executes the graph rather than the file. Terraform builds this graph explicitly and can print it; the details of how edges are inferred and where cycles are rejected belong to resource graphs and dependency ordering. The consequence at this level is that file order and module order carry no meaning. Two resources in the same file with no reference between them may be created in either order, or concurrently.

Concurrency is bounded, not unlimited. Terraform walks the graph with a fixed number of parallel operations, ten by default, which is a compromise between API rate limits and wall-clock time. A graph that is one long chain (each resource depending on the last) gets no benefit from the parallelism setting at all, which is why overly specific depends_on edges are expensive: they serialize work that did not need to be serialized.

What the loop cannot see

The loop compares configuration to a recorded observation, not to the world. Between the last read and the current moment, the world may have changed. The tool’s answer to this is to read again before planning, which is called refresh. Refresh is a full read of every managed object, and its cost scales with the size of the configuration: one API call per resource at minimum, more when the provider must page through children. Large configurations therefore spend most of their plan time on reads, and some tools let the refresh be skipped when the operator knows nothing has changed.

Refresh also cannot see objects that were never recorded. A resource created by hand, or by a different tool, is invisible to the configuration that should own it. The plan will propose creating a second copy, and the API may reject it on a uniqueness constraint or may accept it and leave two objects that look alike. This is the class of problem that adopting existing resources addresses, and it is the reason identity (not configuration) is the hard part of the model.

A third blind spot is state that the tool holds but the API does not. Terraform’s state file records attribute values and dependency edges; it is the only place some of that information exists. Delete it and the objects still exist, but the mapping from configuration addresses to real objects is gone. The file is therefore not a cache that can be regenerated at will, which is why state files get their own storage, locking, and backup story.

Declarative describes objects, not processes

The model’s boundary is the object. Anything that is a sequence of actions on the way to an object (running a schema migration, seeding a database, waiting for a third party to approve a certificate) has no natural representation. Tools provide hooks for these, and the hooks are always the weakest part of the system, because they run once per apply rather than once per convergence, and because their effects are not read back.

Knowing where the boundary sits changes how configurations are written. Work that can be expressed as an object’s attributes converges. Work that must happen in order, or must happen exactly once, does not, and pretending otherwise produces a configuration that appears idempotent and is not. The rest of this cluster is largely about the machinery that makes the object-shaped part trustworthy: the graph, the plan, the state, and the rules for replacing an object when it cannot be changed.