Convergent State Application and Why a Run That Changes Nothing Still Costs Money

A shell script and a desired-state configuration language often execute the same commands. The difference is not syntax. It is that the configuration tool can answer, before touching anything, the question “is this resource already in the requested state?” and can answer it without side effects. Everything else about configuration management follows from how that question is answered.

The gap between a script and a state

Consider an operator who wants a directory to exist with owner www-data and mode 0755. In sh, that is a test and a command:

[ -d /srv/www ] || mkdir /srv/www
chown www-data /srv/www
chmod 0755 /srv/www

This works and is not idempotent in the sense that matters. The mkdir is guarded, so re-running is harmless. The chown and chmod are not guarded, and they do not need to be, because applying the same owner and mode twice leaves the same owner and mode. But the script has no notion of what the resource should look like; it has a sequence of actions that happen to be harmless when repeated. If someone adds a fourth property, a group, an ACL, or a required mount point, the operator adds another line and another guard, and the mapping from “what I want” to “what I do” grows by hand.

A desired-state system splits the request into two separate artifacts. The first is a declaration: this file must exist, be a regular file, be owned by www-data, have mode 0755, and contain these bytes. The second is a provider: code that knows how to read the current state of a file from the system, compare it against the declared state, and emit the minimal set of operations that moves one to the other. The operator writes the first. The tool ships the second.

The critical property is that reading the current state must be cheap and side-effect free. A provider that reads stat(2) and hashes the file body can be asked the question any number of times. A shell command that creates a directory cannot be interrogated; it can only be run.

What the diff is actually comparing

The tool does not compare a desired file with a managed file. It compares the declared attributes with the observed attributes, and only those. This distinction explains most surprising behaviour.

If the declaration says “mode 0644” and the observed file has mode 0640, the diff is non-empty and the tool will call chmod. If the declaration says nothing about mode and the observed file has mode 0600, the diff is empty and the tool does nothing: an undeclared attribute is outside the comparison, not defaulted to zero. This is the same semantics as an absent key in a database row versus a key set to null.

For file contents, naive comparison by reading both sides is impossible, since the desired content may be generated at run time. Tools therefore use a change detector: a checksum of the rendered content stored alongside the managed resource, or the modification time of a source template compared against the target. Both are heuristics. A checksum of the rendered bytes is exact and costs one hash of the file’s length. A timestamp comparison is nearly free but wrong in both directions: it misses a content change that preserves mtime, and it reports a change when a file is touched without modification. Tools that offer timestamp comparison usually expose the checksum comparison as the safe default and document the timestamp mode as a speed trade. What the checksum compares against is itself a resolved value from the layered data, so a reference whose resolution changes can produce a diff with no visible source, which is the problem variable precedence describes.

Templates add a second-order problem. The rendered output depends on variables, which come from other parts of the configuration. A template that renders identically from different inputs is indistinguishable from no change, which is the desired outcome, but it means that changing a variable that does not appear in the template produces no event, and changing one that appears in a comment produces an event that rewrites the file and may restart a service.

Ordering emerges from the graph, not the file

Declarations in a configuration file are written in a sequence, but they are not executed in one. Each resource implicitly depends on the resources it references, and tools add explicit dependency edges in two distinguishable flavours: ordering edges that say “apply this after that”, and notification edges that say “if this changes, tell that”. Puppet separates these into different metaparameters; Ansible encodes ordering in task order plus notify handlers; Chef and Salt mix both models but keep the distinction.

The engine builds a directed graph, checks it for cycles, and applies resources in a topological order. Resources with no path between them may be applied in any order, and in practice are applied in parallel or in hash order. This is why a configuration that works may apply its resources in a different sequence on the next run, and why a hidden dependency on application order manifests as an intermittent failure rather than a consistent one.

Convergence means that the apply phase is repeated until no resource reports a change. A single pass may leave a resource out of sync when an earlier resource in the same pass modified an input to a later one. Most tools run a single pass and report the changes; a few support repeated application until a fixed point. The fixed point is not guaranteed to be reached: two resources that each revert the other’s work produce an oscillation, and the tool will either loop or stop after reporting both as changed on every run.

Property	Scripted run	Convergent apply
Cost of no change	one process per command	one inspection per resource
State read	not modelled	stat, API GET, query
Ordering	textual	dependency graph
Repeat semantics	undefined	should converge

The economics of the no-op run

The no-op run is the ordinary case. Most applies change nothing, and their cost is the cost of inspecting every resource: a stat per file, an API call per cloud object, a package database query per package. On a host with a few hundred resources this is dominated by process spawn overhead, since each inspection is often a separate short-lived process.

For managed cloud resources the inspection cost is a network round trip, and the bill for a no-op run is measured in API calls. A configuration that manages several thousand objects and inspects each one on every run performs several thousand reads per apply. This is why high-frequency convergence on large estates is expensive, and why the drift-detection problem is often solved by collecting state continuously and comparing snapshots rather than by running the full apply loop on a timer.

The reverse side of the same coin: the inspection is the only mechanism that discovers drift. A resource whose state was changed outside the tool is invisible to a system that does not inspect. The choice of inspection frequency is a direct trade between detection latency and read cost, and it is the subject of drift detection and reconciliation loops.

Where convergence stops being free

Convergence assumes the resource can be read and written by the same authority. This assumption fails in several identifiable ways.

Resources without a readable state. Some operations mutate state that cannot be queried: a yum transaction, a database migration that rewrites rows, an imperative API that accepts a “send” command and returns nothing. For these the tool can only guard on a sentinel (a marker file, a version table row) and accept that the sentinel may disagree with reality.

Resources with unmodelled state. A file resource that declares content will rewrite the file whenever the checksum differs, overwriting any edit made by a person or a process. A file resource that declares only permissions will never notice content changes. Neither is wrong; both are consequences of comparing only declared attributes, and mixing the two on the same path in two different declarations produces whichever runs last.

Resources that are not convergent by nature. Package installation resolves to a version, and a declaration of “version 1.4 or later” makes the observed state acceptable at many points, so a newer version already present satisfies the declaration and is left alone. A declaration of an exact version will downgrade when a newer one is present, which is the correct reading of the declaration and rarely the intent.

Resources whose inspection has side effects. Some APIs treat a missing object as an error that must be constructed defensively, and some list operations page through results in a way that costs as much as a write. A provider that inspects by creating a probe object is not idempotent no matter what the declaration says.

The design constraint that forced this shape is worth stating plainly. Configuration management exists because a fleet of machines cannot be kept identical by human discipline, and the only mechanical way to know a machine matches its intent is to compare observed state against declared state. Every property of the tool, from the dependency graph to the cost of a no-op run, follows from that comparison being the primitive operation, and from the fact that the comparison must be exact where the declaration is exact and silent where the declaration is silent.