The Reconciliation Loop and Level-Triggered Control of Declared State

A cluster is a set of machines whose individual fates are not coordinated. Any given node can vanish between two instructions, and no process on the remaining nodes is told that it happened. In that environment, an imperative sequence of commands (create this, then attach that, then start the other) has no recovery point. If the sequence stops halfway, nothing in the system knows what the intended end state was, so nothing can finish it. The alternative is to record the intended end state as data, separately from the machinery that produces it, and run a loop whose only job is to reduce the difference between the two.

That loop is the unit of control in every modern orchestration system. Its structure is small enough to state in a paragraph, and almost every operational property of a cluster follows from that structure: why a half-finished rollout resumes after a restart, why deleting an object is a multi-step protocol rather than a single call, and why two controllers can each be correct while the cluster as a whole is wrong.

Declared state, observed state, and the split that makes the loop possible

Every controlled object carries two halves. The declared half is written by clients and describes what should be true: how many replicas, which image, which resource requests, which placement constraints. The observed half is written by the controller and describes what it last managed to make true. The API server treats them as different resources for write purposes: the observed half is normally exposed as a subresource, so a status write cannot accidentally overwrite a spec write, and a client updating the spec does not have to read-modify-write the status it does not own.

This split is not cosmetic. It gives the loop a fixed point to test against: the loop is finished when the observed half satisfies the declared half, and the test is a comparison of two documents rather than a memory of what was done last time. A controller that restarts has lost all of its memory and none of its inputs, so it can resume from wherever the comparison currently stands. The same property makes a controller safe to run twice, or to run in two copies at once, which is what makes the control plane horizontally replicable.

The loop itself is a work queue fed by a watch, plus a function that takes one key and does whatever is needed for that key. Three details in that sentence matter more than they look.

The queue holds keys, not objects. An object named web-7f9c in namespace prod is queued as the pair (namespace, name), and a second event for the same object while it is still pending collapses into the same entry. This coalescing is what keeps a controller from doing work proportional to the event rate. A controller managing a thousand objects that all change in one second performs at most a thousand reconciles, not the number of watch events that arrived, and if the objects change a thousand times each before the worker picks them up, it still performs one reconcile per key and reads the current state when it does.

The queue is rate-limited per key. A failing reconcile is retried with exponential backoff, and the backoff is reset on success. Without per-key limiting, one object stuck in an unsatisfiable state would spin a worker at full speed and starve every other key in the queue.

for {
    key, shutdown := queue.Get() // block until at least one key is pending
    if shutdown {
        return
    }
    if err := reconcile(ctx, key); err != nil {
        queue.AddRateLimited(key) // requeue with a per-key backoff
    } else {
        queue.Forget(key) // success resets the backoff
    }
    queue.Done(key)
}

And the worker reads from a local cache, not from the API server, for every read. A single watch connection per resource type is shared by all the controllers in the process, and each maintains an in-memory index. Reads against that index cost a map lookup; reads against the API server cost a network round trip and a serialisation pass. A controller doing a hundred reads per reconcile cannot afford the latter.

Why the event is only a hint

The design decision that separates this style of control from an event-driven one is what an event means. Here, an event means “the state of this key may have changed; look again”. It does not carry the change, and the handler does not act on the event’s contents. The controller re-reads the full current object and decides from that.

The reason is that a stream of deltas is not recoverable. If a watch connection drops, and it will, because API server instances restart and network partitions happen, the client has missed an unknown number of changes. Reconstructing the present from a partial delta history requires the history to be complete and ordered, which is exactly what a dropped connection destroys. Re-reading the present requires nothing but the ability to read the present.

So the watch protocol is built around resumption rather than replay. Every object version is a monotonically increasing revision assigned by the underlying store, and a watch request names the revision it wants to start from. If that revision is still within the store’s compaction window, the watch resumes and delivers everything after it. If it has been compacted away, the server answers with HTTP 410 Gone, and the correct client behaviour is to discard the cache and list from scratch. The compaction window is therefore the upper bound on how long a client may be disconnected before it pays a full relist, and clusters that compact aggressively trade disk for relist storms after a network outage.

An edge-triggered design would have to solve a harder problem: it would have to guarantee delivery of every event exactly once and in order, across restarts of both endpoints, or it would silently miss transitions. Level-triggered control replaces that guarantee with a cheaper one: the state is always readable, so any number of missed notifications is repaired by the next read. This is the same distinction as an interrupt handler that reads a status register to find out what happened, versus one that trusts the interrupt cause bits to be complete.

The cost is read amplification. A burst of ten thousand changes to one object produces one reconcile per queue drain, and each reconcile reads the whole object even if one field changed. For objects of a few kilobytes this is irrelevant. For objects that accumulate large lists, such as a status holding thousands of conditions or an object with a very large embedded configuration, the repeated full read becomes the dominant cost, and the usual fix is to split the object rather than to make the read incremental.

Idempotence is a requirement, not a nicety

Because a key can be reconciled at any time, for any reason, including reasons that have nothing to do with a change, the reconcile function must be a no-op when the world already matches. This is stronger than “handle duplicate events”. A periodic resync deliberately replays every key in the cache at a fixed interval, with no change behind it, to repair drift that no event reported: an object deleted by something that bypassed the watch, a status field that a human edited, a downstream system that silently reverted. A controller whose reconcile creates a resource unconditionally will, on the first resync, create a second one.

The practical shape of an idempotent reconcile is: read the object; if it is gone, do nothing; compute the set of children that should exist; list the children that do exist; create the missing ones, delete the extra ones, and update the ones whose content differs; then write status only if the status changed. The “only if changed” clause matters because a status write with identical content still bumps the revision, still emits a watch event, and still re-queues the key. A loop that writes status unconditionally reconciles itself forever at the rate the queue allows.

Ordering inside one reconcile is also load-bearing. Creating a child and then updating the parent’s status in the same pass means a crash between the two leaves the status stale, which is recoverable: the next pass sees the child exists and corrects the status. The reverse order, status first and child second, leaves the status claiming something that is not true, and a reader acting on that status makes a decision on a false premise. The rule that falls out is that observable side effects happen before the record of them.

When the cache is behind the write

The cache is updated asynchronously, so a reconcile that reads an object and then writes it back can lose a concurrent update. The protocol that prevents this is a precondition on the write: the update carries the revision the client read, and the store rejects the write with a conflict if the object has moved on. The client treats the conflict as a signal to re-read and retry, not as an error to surface. This is optimistic concurrency control in the same family as an HTTP entity tag with a conditional request, and it has the same property: no locks are held between the read and the write, so throughput does not degrade with contention, but contention converts into retries.

A controller that ignores the conflict and overwrites unconditionally will pass every single-threaded test and lose writes in a busy cluster. The failure is quiet: the field a human set is replaced by the controller’s stale copy, the human’s next write is accepted, and the two oscillate. The symptom is an object whose spec changes without anyone changing it, which is the signature of a lost-update loop between two writers.

Startup has its own hazard. A controller must not begin processing until its cache has completed its initial list, because before that point every read returns “not found” and every reconcile concludes that the object must be created. The standard shape is to block the workers until the initial synchronisation completes, then start them; the queue buffers events during that window. Skipping the wait produces duplicate children at every restart, in proportion to how many objects the controller manages.

What the loop cannot promise

Reconciliation converges, but it does not converge in bounded time, and it does not converge at all under some conditions. There is no global transaction spanning controllers: each one sees a snapshot of its own inputs, acts, and the next one acts on the result. Two controllers can therefore each be locally correct and jointly wrong, with the wrongness expressed as an oscillation that neither can see. The scheduler’s placement decision and the controller that creates the workload are a concrete pair: the controller creates a workload that the scheduler cannot place, and the loop retries forever without any component reporting an error, because from each side’s view nothing has failed. The remedy is not a stronger loop but a constraint that makes the request satisfiable, a subject treated in placement constraints and unschedulable states.

The other limit is that the loop can only control what it can observe. A controller that assumes a node’s kubelet will honour a request has no way to detect that it did not, unless something reports back. The observed half of the object is that report, and it is only as fresh as the last heartbeat, which is why liveness is itself a leased, time-bounded protocol, as described in heartbeats, leases, and node failure detection.