Idempotency Keys and Command Resources That Cannot Be Described Declaratively

Declarative configuration models a resource as a set of attributes with a known state. That model covers files, users, packages, services, and most cloud objects, because each of those has a readable state and a defined transition. It does not cover the class of operations that exist only as a verb: applying a database migration, issuing a certificate for a hostname, seeding a search index, running a one-off schema fix. These have no observed state to read, only a history of having happened.

The standard workaround is a command resource with a guard. The guard converts “run this” into “run this unless it appears already done”, and the appearance of done is a predicate the operator writes. This is the same check-then-act shape that convergent state application produces mechanically for resources that have readable attributes; the difference is who supplies the comparison. This article is about how that workaround works, and about the specific ways it fails.

Writing the guard, and what it can observe

A command resource runs a program. The guard is a second program whose exit status decides whether the first runs. Most tools express it as a shell test or as a small script whose zero exit means “skip”.

# A guard that asks the database itself, rather than a local marker
applied=$(psql -tAc "select count(*) from schema_migrations where version = '0042'")
if [ "$applied" -ne 0 ]; then
  exit 0        # already applied, skip the migration
fi
exit 1          # not applied, run it

The guard’s exit status is a single bit, and that bit is the entire state model for the resource. Everything the tool knows about the migration is whether the guard returned zero. There is no diff, no attributes, and no report of what differs. The consequence is that the tool cannot tell “not applied” from “applied incorrectly”; both are the same bit, and the only available action is to run the command again.

Two choices matter here. The guard can read the system of record (the migrations table above), or it can read a local artifact (a marker file in a well-known path). A system-of-record guard is authoritative and shared across hosts, so re-running on a fresh machine behaves correctly. It costs a query per run and requires the query itself to be cheap. A marker file is free and local, and it becomes stale the moment the underlying system is restored from a backup, moved, or reset. Marker files are the wrong choice for anything whose state lives elsewhere, and the right choice for operations that touch only the local machine.

The correct direction of authority is the deciding rule: the guard should read the same source that the operation writes. A migration writes rows to a schema table, so the guard reads that table. A certificate request writes an object in a certificate store, so the guard reads the store. A local build writes files under a prefix, so the guard can test the prefix.

Why an ordered sequence needs an identifier, not a boolean

A boolean guard is wrong for multi-step processes. Consider three migrations, 0040 through 0042, each with its own guard testing its own version. A host where 0042 is applied but 0041 is not cannot exist if 0041 and 0042 were applied in order by the same tool, but it can exist after a partial failure or a manual intervention. Every per-step guard then reports zero for 0042 and non-zero for 0041, and the tool applies 0041 and then skips 0042. Whether that is coherent depends entirely on whether the migrations are individually reversible and order-independent, which is a property the tool cannot check.

The fix is to encode a position rather than a set: a single integer marking the highest applied step, with the guard comparing against it. The operation becomes “apply everything above the recorded position”, which is a monotone sequence and therefore has a well-defined result after any prefix of it has succeeded. This is why migration tools track a single version and why configuration systems that drive migrations should let the migration tool own that version, rather than re-deriving it from a set of files present on disk.

Sets have a second failure mode: subtraction. A file present in the configuration source is applied; a file deleted from the source is not removed unless the tool is told to purge. The migration analogue is a step removed from the list, whose effects remain in the database. There is no declarative remedy for this, because the effects have no attribute to compare. The remedy is append-only discipline: steps are added, never removed, and a correction is a new step.

A command resource is also the one place where an apply can fail for a reason the declaration cannot describe, and the shape of that failure is covered in dependency ordering and partial failure: the guard’s bit satisfies every downstream dependency, so a wrong guard shows up as a missing prerequisite two resources later.

The race between guard and command

A guard and its command are two separate processes with a gap between them. Two concurrent applies on the same host, or two hosts sharing a database, can both pass the guard before either runs the command.

The size of the gap is bounded below by process creation. Each side of the guard is at minimum one fork and one exec on a POSIX system, which costs on the order of a millisecond on ordinary hardware before the program has done anything. An operation whose guard query is a local read therefore has a window of roughly a millisecond between check and act. An operation whose guard crosses a network has a window equal to the round-trip time plus the scheduling of both processes, which on a shared database is tens of milliseconds and occasionally seconds under lock contention. Nothing in the tool narrows this window, because the tool does not know which resource the guard protects.

The gap is small but not atomic, and the failure it produces is not a spurious run: it is a duplicated application. For a migration that creates a table, the second run fails on the existing table and the tool reports a failed resource. For a migration that inserts a row, the second run inserts a duplicate, and the tool reports success. The second case is the dangerous one, and it is invariant to how tight the gap is: the only defenses are a uniqueness constraint at the sink or a lock taken before the guard is evaluated.

Tools that support concurrency within a single host usually serialize resources with an ordering edge, so the intra-host race does not occur there. The inter-host race is a property of the shared system, not the configuration tool, and it must be solved with a lock in the shared system. PostgreSQL advisory locks, a SELECT ... FOR UPDATE on the version row, or an atomic insert into a uniquely constrained table all work; which one is appropriate depends on the transaction boundaries of the operation itself.

Locks reintroduce a cost that the guard was meant to avoid. A lock held across a long migration serializes the fleet and converts parallel convergence into a queue whose length is the number of hosts. A lease with a timeout bounds the damage from a holder that dies without releasing, at the price of admitting a second holder when the first is merely slow. Both are correct only if the migration commands themselves tolerate being attempted twice, which pushes the problem back to the sink: the durable answer is a unique constraint or a conditional write at the place where the effect lands, and the lock is an optimization that reduces wasted attempts rather than a correctness mechanism.

Modelled state versus observed state

A useful exercise is to write down, for any command resource, which of three things the declaration is actually asserting.

Modelled state: the guard’s query is exact and cheap, and the command’s effect is fully captured by it. A row in a table that records the applied version is such a case; the guard and the effect are the same object, so the declaration is as strong as a file resource’s checksum.

Approximate state: the guard tests a consequence of the effect rather than the effect. Testing that a directory exists after an unpack, or that a process responds on a port after a service start, is this case. It is usually good enough, and it is why service resources in most tools expose a liveness check rather than merely a recorded intent. The approximation breaks when the consequence is achievable by a different cause, at which point the guard reports done and the real work silently never happens.

No state: the effect is a notification or an external side effect with no queryable residue. Sending a webhook, issuing a refund, embedding a line into a third-party system that exposes only a write API. These cannot be made idempotent by the configuration tool at all. The honest handling is either to route them through a system that deduplicates on an identifier carried by the request, or to accept that the operation is not convergent and keep it out of the regular apply loop.

The third row is the boundary of the whole approach. Configuration management guarantees convergence on resources whose state can be read, and for everything else it offers a convention. The convention propagates: a command resource with an approximate guard is a hidden dependency of everything ordered after it, because the dependency is satisfied by the guard’s bit, and that bit is what the next resource waits on. When the guard is wrong, the failure appears downstream as a missing prerequisite in a resource that did its own job correctly, and the reported error names the wrong resource. This is the single largest source of confusing failures in otherwise well-structured configuration trees, and the remedy is structural: keep the number of unmodelled operations small, give each one a guard that reads the system it writes, and treat the guard’s query as part of the interface rather than as a shell-wrapper detail.