Level-Triggered Reconciliation and the Desired-State Loop

A command interface and a declarative interface look interchangeable until the first failure. create-volume --size 100 either runs or does not; if the client dies between sending the request and reading the reply, nobody knows which. A declaration — a record saying a volume of that size should exist, owned by this service — survives the death of the client, because the record is the input to a loop that runs elsewhere and keeps running. Every self-service platform is built on that difference, and the loop is the mechanism that makes it true.

The loop is small: read what should be, read what is, compute the difference, take the action that closes it, then repeat. The interesting engineering is in the four words “read what is”, because the naive reading of that phrase produces a system that is either wrong or unusable.

Why the loop is level-triggered rather than edge-triggered

An edge-triggered design reacts to events: a creation event arrives, so create the dependent object; a deletion event arrives, so delete it. It is efficient, and it is fragile in one specific way: an event that is dropped, or handled by a process that crashes before it finishes, leaves no trace that work remains. Recovery requires an external log of unhandled events, which is exactly the state the design was trying to avoid.

A level-triggered design treats events as hints. The event only says “something in this neighbourhood may have changed, look again”. The handler then reads the complete current state of the objects it cares about and compares it to the desired state, without reference to which event fired or how many arrived. Losing an event costs latency, not correctness, because the next event, the next periodic resync, or the next restart of the controller produces the same comparison.

This is the property that lets a platform survive its own restarts. A controller that has been down for an hour does not need to replay an hour of events; it starts, lists everything, and reconciles each object once. The reconciliation function must therefore depend only on the pair (desired, observed) and never on history. Anything the function needs to remember between passes belongs in the API object itself, usually in status, not in the process’s memory.

The same discipline explains why declarative systems converge under repeated application. Applying the same declaration twice produces the same result, because the second pass computes a zero difference and does nothing. Idempotence is not a stylistic preference here; it is the property that makes retries, resyncs, and duplicate event delivery harmless.

The four steps and where each one breaks

Observe. The controller lists or watches the objects it owns. Watches deliver a stream of changes from a starting resourceVersion, and the API server may refuse a start point that is too old, returning 410 Gone because the change history it would need has been compacted. A correct client treats that as a signal to re-list from scratch, not as a fatal error. The list is the authoritative snapshot; the watch is an optimisation on top of it.

Diff. The comparison is between the declared spec and the reported status, plus whatever live state exists outside the API server. For a cloud resource, the second half of the comparison requires a call to the provider’s API, which is where reconciliation gets expensive and where rate limits enter the design.

Act. Actions must be expressed as idempotent requests. “Create the object named X” is safe to repeat if the second attempt observes the existing object and adopts it. “Append a rule to the security group” is not, and repeated reconciliation will append it repeatedly. The practical rule is that every action either sets an absolute value or checks for existence first.

Requeue. After acting, the controller does not assume success. It requeues the object, either immediately or after a delay, and the next pass observes whether the action took effect. This is why controllers appear to do everything twice: the first pass writes, the second pass confirms.

func (r *Reconciler) Reconcile(ctx context.Context, req Request) (Result, error) {
    var obj Example
    if err := r.Get(ctx, req.NamespacedName, &obj); err != nil {
        // Deleted between queueing and handling: nothing to do, and not an error.
        return Result{}, client.IgnoreNotFound(err)
    }
    if !obj.DeletionTimestamp.IsZero() {
        return r.finalize(ctx, &obj)
    }
    // Compare full desired state, not the event that woke us.
    if diff := desiredVsObserved(&obj); diff.Empty() {
        return Result{}, nil
    }
    if err := r.apply(ctx, &obj); err != nil {
        // Backoff is per key, so one broken object cannot starve the others.
        return Result{}, err
    }
    return Result{Requeue: true}, nil
}

Status as a second channel, and generation as the guard

The spec is written by tenants and by the platform’s own templates; the status is written by the controller. Splitting them into separate subresources means a status update cannot clobber a concurrent spec update, and a tenant editing the spec cannot forge a status that says the change succeeded.

The generation field increments on every spec write, and the controller copies the generation it has processed into status.observedGeneration. A consumer that reads status.ready: true without checking that observedGeneration matches generation is reading a verdict about an older version of the object. This is the single most common source of “the platform said it was ready and it was not”: readiness was reported for the previous spec, and the new spec has not been reconciled yet.

Updates use optimistic concurrency. Every object carries a resourceVersion, and a write that includes a stale one is rejected with 409 Conflict. Controllers therefore read, modify, and write with the version they read, retrying on conflict. A controller that ignores conflicts and writes unconditionally will silently overwrite fields owned by another actor, which is the mechanism behind a class of bugs where a manually applied annotation keeps disappearing.

Ownership, deletion, and the objects nobody owns

Reconciliation creates objects, and creation raises the question of who may delete them. Kubernetes answers with ownerReferences: a dependent object names its owner by UID, and the garbage collector deletes dependents when the owner is gone. The UID matters — a reference by name alone would attach the dependent to a newly created object that happened to reuse the name, which is the wrong object.

Deletion of a platform-managed resource is where the loop most often stalls. A finalizer is a string in metadata.finalizers that keeps an object in the API server after deletionTimestamp is set, so a controller can release external resources first. If that controller is not running, or its finalizer logic has a bug, the object stays in a terminating state indefinitely, and every dependent object that waits for its removal waits forever. A platform that provisions cloud resources should assume finalizers will occasionally strand objects and provide a documented way to inspect and clear them, because the alternative is a tenant with a resource that cannot be deleted and cannot be replaced.

Ownership also decides the direction of enforcement. An object created by the platform from a tenant’s declaration is managed: if a human edits it, the next pass reverts the edit, and the human sees their change disappear. That is correct behaviour for a generated object and infuriating behaviour for one the tenant believes they own. The boundary has to be explicit in the platform’s interface, not discovered by experiment.

The costs that decide whether the design is usable

Level-triggered reconciliation buys correctness with traffic. Each pass reads state, and the number of passes is roughly the number of objects times the event rate plus the resync rate. A controller watching ten thousand objects with a resync of ten minutes issues on the order of a thousand list or get operations per minute even when nothing changes. The API server’s watch cache absorbs much of this, but a controller that lists the whole collection inside every per-object reconcile defeats the cache and scales quadratically with object count.

Two failure shapes recur. The first is the hot loop: a reconcile that returns an error for a condition it cannot fix, with no backoff, produces an unbounded retry storm against the API server or the cloud provider. Per-key rate limiting and exponential backoff with a cap are not optional. The second is the thundering herd: a controller restart causes every object to be queued at once, and if each reconcile calls an external API, the provider’s rate limit is hit immediately. Jittered requeues and a bounded number of concurrent workers spread that burst.

The deeper limitation is that reconciliation is not a transaction. Two objects can each be reconciled against a consistent view and still leave the pair in a state that violates a rule relating them, because no pass ever sees both mid-update. Rules that span objects cannot be enforced by a per-object loop; they need either an aggregated object that the controller owns as a unit, or an admission-time check that sees the object at write time — the subject of what an admission webhook can decide. Deciding which of those two a given rule needs is most of the design work in a platform’s control plane, and it is the same decision that reappears when the desired state itself lives in Git, as described in sync ordering and pruning.