Level-Triggered Reconciliation: Comparing State Instead of Consuming Events

A control loop that keeps a distributed system in a desired state has two possible inputs: notifications that something changed, or a periodic look at what the state actually is. The first is cheaper and is what most systems reach for. The second is what makes the system converge after the failures that matter, because the events that would have driven the first are exactly the ones lost when a component crashes, a queue drops a message, or a watch expires. The distinction between edge-triggered and level-triggered control is the difference between a system that reacts and a system that converges.

Edge-triggered control and the events it loses

An edge-triggered loop subscribes to changes and acts on each one. A deployment controller watches for a new desired replica count and starts the missing instances; a rebalancer watches for a node joining and moves data to it. Each event carries the delta, and applying the delta is the whole of the work.

The design is efficient because it does nothing when nothing changes, and it is fragile for a reason that has nothing to do with the code: the notification channel is itself a distributed system, and it fails independently of the thing it reports on. Three losses are common.

The first is a missed event during a crash. The controller is down when the event is published; the event is delivered to nobody or dropped after a timeout. On restart the controller has no record of the change and does nothing.

The second is a coalesced or reordered event. A system that publishes “count changed to 5” and then “count changed to 6” while the consumer is stalled may deliver only the latest, which is fine for a value but wrong for an increment. A system that publishes “a node joined” and “a node left” may deliver them out of order, and applying them in the wrong order leaves the wrong membership.

The third is the event that never existed. Some state changes have no event at all: an instance dies without notifying anything, a disk fills, a process becomes wedged but does not exit. There is no publisher for these, so no subscription can observe them. Any controller that relies solely on notifications is blind to precisely the failures it exists to repair.

The consequence is that an edge-triggered loop drifts. Each lost event leaves a permanent divergence, and the divergences accumulate over the lifetime of the system. The system appears correct after each individual operation and becomes progressively wrong in aggregate, which is the hardest kind of failure to attribute.

Level-triggered comparison

A level-triggered loop inverts the relationship. Instead of being told what changed, it reads the current state, compares it against the desired state, and computes the difference itself. The action is the same whether the divergence arose from a lost event, an unannounced failure, or a deliberate change. The loop does not know or care about the cause. It knows only that observed differs from desired, and it takes a step toward closing the gap.

This is the mechanism Kubernetes describes as a controller: an object records the desired state, the controller observes the actual state, and it acts to move one toward the other. The observation is periodic and unconditional. A crashed controller restarts, observes, and resumes; the only thing it lost was time. A missed event costs nothing because events are not the input.

// Reconcile drives observed toward desired without knowing what changed.
func (r *Reconciler) Reconcile(ctx context.Context) error {
    desired, err := r.store.Desired(ctx)
    if err != nil {
        return err // retry with backoff; do not guess
    }
    observed, err := r.cluster.Observe(ctx)
    if err != nil {
        return err // an unobservable cluster is not an empty one
    }
    for _, id := range missing(desired, observed) {
        if err := r.cluster.Create(ctx, id); err != nil {
            return err // partial progress is fine; next pass resumes
        }
    }
    for _, id := range extra(desired, observed) {
        if err := r.cluster.Delete(ctx, id); err != nil {
            return err
        }
    }
    return nil
}

Two properties of that shape carry the correctness. The first is that a partial pass is safe: the loop may create three of five missing instances and fail on the fourth, and the next pass observes the three it created and computes a smaller difference. There is no per-operation state to lose, because the state is in the cluster and the loop reads it. The second is that errors must not be treated as absence. A failed observation that is interpreted as “the cluster is empty” causes the loop to create everything again, which on a large system is a destructive action taken in response to a transient network error.

The cost of looking

Full comparison is not free. Observing every object in a large system on every pass costs a query proportional to the size of the state, repeated at the loop’s period, and the cost grows with the system rather than with the change rate. On a cluster with a hundred thousand objects and a reconciliation period of thirty seconds, the loop issues a few thousand object reads per second continuously, most of which find nothing to do.

Four techniques keep this affordable, and they compose.

Caching the observation and invalidating it on events combines the two models: the loop reads the full state periodically but refreshes its cache when a change notification arrives, so the common case is a cache lookup and the periodic read is the safety net. The event path improves latency; the periodic path provides convergence. This is the standard arrangement, and the important detail is that the periodic read must be genuinely independent: it cannot be skipped because the cache is believed fresh, since the belief is exactly what a missed event invalidates.

Partitioning the state by key range lets independent workers reconcile disjoint slices in parallel, which converts a linear cost into a cost per worker. The partitioning must be stable enough that each slice is owned by exactly one worker; a slice reconciled by two workers concurrently will issue duplicate creates, which is why create operations must be idempotent by name rather than by absence check.

Generation numbers avoid the comparison entirely in the steady state. Each desired-state object carries a generation that increments on every change, and the observed state records the generation it was reconciled from. A pass over a range of objects reads only the generations, not the objects, and reconciles only those whose generation is behind. The read is still proportional to the number of objects, but it is a read of small metadata rather than of full specifications.

Rate limiting the diff, rather than the observation, bounds the blast radius of a mass divergence. If a bug or a bad configuration makes every object diverge at once, an unbounded loop attempts every repair simultaneously and turns a control-plane problem into a load problem. Applying repairs at a bounded rate keeps the system responsive while it converges, at the cost of a longer convergence time after a large change.

Convergence is not immediate and cannot be assumed

A level-triggered loop converges eventually, and the word eventually is doing real work. Between the moment a desired state is written and the moment the observed state matches it, the system is in an intermediate condition, and every consumer of that state must tolerate it. A deployment that declares success when the desired state is written, rather than when the observed state matches, reports success before any instance exists.

The loop’s period sets a floor on detection latency: a divergence that appears just after an observation waits nearly a full period before being seen. A period short enough to detect failures quickly is a period short enough to make the observation cost significant, and the trade is between the two. Most systems resolve it by making the period adaptive: short when the state is changing, long when it is quiet. That requires the loop to know whether it is in a transient, and the honest way to know that is to track how many objects diverged in the last pass.

Convergence also fails outright under oscillation. If the loop’s action changes the observation in a way that makes the next pass act in the opposite direction, the system never settles. The classic form is a comparison with a tolerance: an autoscaler that adds an instance when utilisation exceeds 70 percent and removes one when it falls below 65 percent will, with a workload that lands between them, add and remove forever. The fix is either a dead band wide enough to absorb the loop’s own effect, or hysteresis that makes the remove condition depend on a state the add condition cannot produce. The same instability appears in the background-work controller described in admission control for background jobs, where it is the gain rather than the dead band that is tuned.

Multiple reconcilers and the ownership question

Once more than one loop can act on the same objects, the model needs an ownership rule. Two loops that both believe they own a set of instances will fight: one creates, the other deletes, and the observed state oscillates between two desired states that are each internally consistent.

Three arrangements are workable. Ownership can be encoded in the desired state itself, so that an object declares which controller manages it and every other controller ignores it. Ownership can be exclusive by construction, so that the sets of objects the loops can touch are disjoint. Or a single loop can own the whole state and delegate the actual work, which keeps the comparison in one place at the cost of a bottleneck.

What does not work is deriving ownership from observation. A loop that claims whatever it can see will claim objects belonging to another loop as soon as that loop is briefly unavailable, and the conflict appears later, when the other loop returns and finds its objects changed. The ownership must be recorded before the object exists, which is why the desired-state store, not the cluster, is the source of truth in this design.

Where level triggering is the wrong tool

Full comparison is the right default for state that is small relative to the system, slow-changing, and idempotently repairable. It is the wrong tool for state that is large and fast-changing, where reading everything to find a small difference costs more than the difference is worth: a per-request cache, a high-rate event stream, a queue of short-lived messages. Those need incremental processing, and they accept the risk of drift because the drift is bounded by the data’s own lifetime.

The dividing line is whether the state can be reconstructed from a durable record. A controller’s desired state can, because it is stored. A cache cannot, and does not need to be: an entry that was never populated is a miss, which is a correct outcome. The systems that get into trouble are the ones in the middle: state that is expensive to reconstruct and too large to compare in full. For those the answer is usually a hybrid: incremental processing for the common path, plus a periodic audit over a sample or over a summary, which is the same reasoning that makes range digests a viable substitute for full comparison.