Reconciliation Loops: Why Level-Triggered Control Beats Event Handling

An internal platform is a program that continuously makes a fleet match a description. The description is stored; the fleet is observed; the difference is closed. Every platform eventually has to choose how that closing is driven, and the choice determines what happens when a notification is lost, when a controller crashes mid-operation, and when two actors change the same object at once. The two candidate designs are event-triggered and level-triggered, and the difference is not stylistic.

Event handling loses information; comparison does not

An event-driven design reacts to change notifications. A watcher subscribes, receives a stream of “object created”, “object modified”, “object deleted”, and dispatches a handler per event. It is efficient, because work happens only when something changes, and it is fragile in three specific ways.

The first is loss. A watcher’s connection drops, or the buffer between the API server and the watcher overflows because the handler is slower than the change rate, and the stream is truncated. The API server signals this by closing the watch, and a well-written client re-establishes it and relists. A client that treats the stream as reliable simply misses events and never learns that it did.

The second is ordering. Events for the same object arrive over a single stream in order, but events for related objects do not, and neither do events across a resync boundary. A handler that receives “pod deleted” before “pod created” for the same name, because the create happened during a reconnect gap, will take an action based on a state that never existed.

The third is the partial operation. A handler that creates three objects and crashes after two leaves the system in a state no event describes, because no event was emitted for the missing third.

Level-triggered control inverts all three. The controller does not consume events as instructions. It observes the current state, compares it to the desired state, and computes the difference. Events, when used at all, are hints that a comparison is due; they are never the source of truth. A missed event delays a reconciliation until the next periodic pass. A crash mid-operation is repaired on the next pass, because the next pass re-derives the whole difference from observation rather than from memory of what it was doing.

The shape of a reconcile function

The contract is small: given the name of an object, read the desired state, read the observed state, act to reduce the difference, and report whether the difference is closed.

func (r *Reconciler) Reconcile(ctx context.Context, req Request) (Result, error) {
    var want appv1.Service
    if err := r.Get(ctx, req.NamespacedName, &want); err != nil {
        if apierrors.IsNotFound(err) {
            return Result{}, nil // nothing to do; garbage collection handles children
        }
        return Result{}, err
    }

    observed, err := r.observe(ctx, &want)
    if err != nil {
        return Result{}, err // requeue with backoff; do not guess
    }

    if diff := diffState(want.Spec, observed); diff.Empty() {
        return Result{}, nil // steady state, no writes
    } else if err := r.apply(ctx, &want, diff); err != nil {
        return Result{}, err
    }

    // re-observe before claiming convergence; the write may not have landed
    return Result{Requeue: true}, nil
}

Three properties of that function carry the design. It returns without error when the object is gone, so deletion is handled by ownership and garbage collection rather than by explicit teardown logic. It returns the error rather than swallowing it, so the work queue can apply exponential backoff. And it requeues after a successful write instead of assuming the write achieved the desired state, because the write is a request to a system that may reject it, may be racing another writer, or may have accepted it partially.

The re-observe step is where many controllers are subtly wrong. After writing, they mark the object as reconciled in memory. The next pass then compares against a stale cached copy and concludes that everything is fine. Comparing against a fresh read costs one API call per pass and removes the whole class of bugs where the controller’s belief and the API server’s state diverge.

Idempotency at the level of the operation

Level triggering gives idempotency at the level of the whole pass for free, provided the individual operations are also idempotent. Creating an object that already exists must be treated as success, not as a conflict to be logged and dropped. Deleting an object that is already gone must be treated as success. Applying a patch that changes nothing must be a no-op.

The mechanism that makes this practical is server-side apply or an equivalent merge: the controller submits the fields it owns, the server merges them with fields owned by others, and the result is deterministic regardless of how many times it is submitted. Without a merge protocol, a controller that reads, modifies, and writes the whole object will fight any other controller that does the same, and the two will overwrite each other’s fields in a loop that produces continuous writes and no progress.

Field ownership is the detail that makes the merge tractable. Each field in the stored object records which actor set it. A controller that owns spec.replicas can change it; a controller that does not own it and tries to change it gets a conflict rather than a silent overwrite. This turns the multi-writer problem from a race into a reported disagreement, which is the only form in which it can be debugged.

Oscillation, and the conditions that produce it

A reconciliation loop converges when the difference it computes decreases monotonically. It oscillates when two actors compute differences from different premises and each undoes the other. Three patterns produce this.

Two controllers claiming the same field. One sets a default of three replicas, the other sets a default of two, both on objects they do not own exclusively. Each pass flips the value, and the loop generates a write per pass forever. The fix is ownership, not arbitration.

A controller whose desired state depends on its own observed output. A controller that sets a label to the hash of the object’s current content will change the content by setting the label, which changes the hash. The remedy is to compute the hash over the fields the controller does not write, or to store the hash in an annotation excluded from the hashed set.

A status writer that conflicts with a spec writer on the same object. If the controller writes status and spec in one update, and another actor writes spec, the status write fails with a conflict and is retried; if the retry recomputes status from a spec that just changed, the status churns. Separating the status subresource from the spec update makes the two writes independent, which is why the API exposes them separately.

Detecting oscillation in practice requires counting writes per object over a window. A controller that writes to the same object more than a small number of times without the object’s generation changing is not making progress, and the platform should report that as an error rather than continuing quietly.

The cost of polling

Level triggering has a price: API traffic proportional to the number of objects times the pass frequency, whether or not anything changed. A fleet of ten thousand objects reconciled every minute produces ten thousand reads per minute even in a perfectly steady state. Two mechanisms reduce this without giving up correctness.

Informers maintain a local cache fed by a watch, so the read half of the comparison is served from memory. The controller still compares full state, but it does not pay an API round trip per object per pass. The cache is eventually consistent, which is acceptable because the controller’s writes are guarded by the API server’s conflict detection; a stale read leads to a rejected write, not a corrupt state.

Work queues deduplicate. When an object changes, its key is added to a queue that holds at most one entry per key. Ten changes to one object produce one reconciliation. This decouples the pass rate from the change rate and collapses bursts.

The residual cost is the periodic full resync, which exists precisely to repair drift the watch missed. Its interval is a trade: shorter intervals bound the staleness window at the cost of traffic, longer intervals reduce traffic and extend the window in which a missed event leaves the fleet wrong. An interval on the order of minutes is common because the failures it repairs are rare and not urgent.

When the loop should refuse to act

The hardest case is the object that cannot be reconciled. A referenced secret does not exist. An image cannot be pulled. A required node pool has no capacity. The loop retries with backoff, which is correct, but it must also report the condition rather than retrying silently. A condition on the object’s status with a reason and a timestamp is the difference between a platform that is debuggable and one where a tenant sees a pending workload and no explanation.

Backoff itself needs a bound. Exponential backoff without a cap produces intervals long enough that a transient failure recovered in seconds is not retried for hours. A cap in the low minutes keeps recovery responsive while still protecting the API server from a fleet of failing controllers. The interaction between reconciliation failure and the platform’s error budget is where this mechanism meets service level objectives as a platform promise.