Confirmed Commit and Automatic Rollback for Remote Configuration Changes

A configuration change that removes the path used to deliver it is self-locking. The change is applied, the session that made it dies, and the only way back is the mechanism the change just disabled. Every scheme for surviving this has the same shape: the change is applied provisionally, a timer runs, and unless something confirms the change before the timer expires, the device reverts to the state it held before. The interesting part is not the idea but the conditions under which the confirmation can be delivered.

Two datastores and one commit

The model that makes this workable separates the configuration being edited from the configuration in effect. A candidate datastore holds edits that no forwarding decision reads. A commit operation validates the candidate and copies it into the running datastore, at which point the change takes effect. A confirmed commit adds a third element: the commit is provisional for a bounded interval, and the device keeps a copy of the previous running configuration so that it can be restored.

The protocol that specifies this in detail is NETCONF, which defines a :candidate capability for the separate datastore and a :confirmed-commit capability for the timer. The confirmed commit operation carries an optional timeout, and the device starts a rollback timer when the commit is applied. If a confirming commit arrives before the timer expires, the provisional state becomes permanent and the saved previous configuration is discarded. If the timer expires without a confirmation, the device restores the saved configuration and the change is undone.

The timeout is the safety margin for the operator’s own access. A change that breaks management connectivity is confirmed only if the operator can still reach the device, so the timeout must exceed the time needed to observe the effect and re-establish access. The protocol’s default, when the parameter is omitted, is 600 seconds, and the mechanism is useless if the timeout is shorter than the time to reconnect.

<!-- provisional commit: revert after 120 seconds unless confirmed -->
<rpc message-id="101"
     xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <commit>
    <confirmed/>
    <confirm-timeout>120</confirm-timeout>
  </commit>
</rpc>

<!-- the confirmation: same operation without the confirmed flag -->
<rpc message-id="102"
     xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
  <commit/>
</rpc>

The two messages differ by one element, and that element is the entire difference between a change that persists and one that expires.

Why a rollback needs the pre-change configuration, not a diff

The device must be able to restore the exact state it held, which means it must have a complete copy of the running configuration from before the commit rather than a record of what changed. A diff is not sufficient because applying the reverse of a diff depends on the rest of the configuration being unchanged since the diff was taken, and a second change made during the confirmation window breaks that assumption.

Storing a full copy also makes the rollback independent of validation. Restoring the saved configuration is a copy, not a re-validation, so it succeeds even if the change being undone introduced a state that the current validation rules would reject. That property is what makes rollback usable as a recovery mechanism rather than as another change that might fail.

The cost is memory and commit latency proportional to the size of the configuration. A device with a large configuration keeps two full copies during the confirmation window, and the copy is what makes the commit operation slower than a plain write. On devices where the configuration is large enough for this to matter, the confirmation window is a period of elevated memory use, which is worth knowing before scheduling a change on a device that is already near a limit.

Confirmation must not depend on the path being changed

The mechanism fails in a specific and predictable way when the confirmation is delivered over the same path the change affects. If the change removes the management interface’s address, or replaces the certificate the management session authenticated with, or alters the access control rule that admits the management source, then the confirming commit cannot be sent. The rollback timer then fires and the device reverts, which is the correct behaviour and also means the change can never be made in one step.

The standard resolution is a two-phase approach: make the change in a form that leaves a usable path, confirm, then make the final restrictive change under a fresh confirmation. That second step has the same problem, so operators often leave a dedicated out-of-band path untouched by configuration changes and confirm through it. The out-of-band path is the one piece of the design that must not be part of the change being tested.

A subtler version of the same failure arises from state that survives the rollback. The rollback restores configuration, not runtime state. A routing adjacency brought down by the change is restored by the configuration reverting, but the adjacency needs to re-form, and the timer that governs re-formation may outlast the confirmation window. A session that was flushed by an access control change may not resume immediately. The device is configured correctly after rollback and is not yet behaving as it did, which is why the confirmation should test behaviour rather than only the presence of a management session.

Rollback is not a substitute for validation

A device that validates a candidate before committing catches errors that make the configuration internally inconsistent: an address on an interface that does not exist, a route pointing at an undefined next-hop group, a policy referencing a filter that was deleted. Validation does not catch a configuration that is internally consistent and externally wrong, such as an address that conflicts with a peer, or a routing policy that is syntactically valid and drops the prefixes it was meant to admit.

Those are the cases the confirmation window exists for, and they require a behavioural test: a probe along the path the change was meant to affect, a check that the adjacency is up, a query of the routing table for a prefix that should be present. The confirmation is the assertion that the test passed, and the timer bounds how long the device waits for that assertion.

The failure mode to guard against is confirming on the basis of management reachability alone. A device whose management plane is reachable can still be forwarding nothing, and confirming that state converts a reversible change into an outage. The confirmation should be tied to the property the change was intended to produce. A change to a routing policy is the clearest case, because it can leave a session established and carrying no routes at all, a state that reading a BGP session failure distinguishes from a transport fault by looking at prefix counts rather than at the session state.

Concurrent changes during the window

The confirmation window is a period in which the running configuration is provisional, and a second change made during it is interpreted against the pending commit rather than treated as independent.

The first case is the one that matters most: a <commit> without the <confirmed> parameter is, by definition, the confirming commit. Whatever else it carries, it ends the provisional state and makes the earlier change permanent. A session that issues an ordinary commit during another session’s window therefore confirms a change it was not necessarily asked to approve, which is why the window needs an owner rather than a convention.

A second confirmed commit behaves differently. The timer is reset to the new interval, and the state restored on a timeout remains the one that preceded the first commit, so a rollback undoes both changes. The same section of the specification makes the session binding explicit: unless the original commit carried the persist parameter, both the follow-up and the confirming commit must be issued on the session that issued the confirmed commit, and a session that terminates before the timer expires forces the rollback.

For a managed fleet, the safe rule follows from those semantics: a device in a confirmation window accepts no other configuration change. A management system can enforce this by holding a per-device lock, which is the same lock that prevents two concurrent editors from overwriting each other’s work. The lock is not part of the confirmed-commit mechanism, but the two are used together for the same reason: the window is a period during which the device’s state has an owner.

What the mechanism does not cover

Rollback restores configuration. It does not restore state that the configuration generated: counters reset, log buffers continue, and any traffic that was dropped during the failed change stays dropped. For a change that affects forwarding, the outage lasts from the moment the change took effect to the moment the rollback completes, and the rollback itself takes time proportional to the configuration size.

Nor does the mechanism help when the failure is not on the device being changed. A change that misconfigures a routing policy on one router can break connectivity between two other routers, and the confirmation from the changed device’s own management path will succeed because that path is unaffected. Establishing that the change was bad requires an observation somewhere else in the network, and that observation is what the confirmation is asserting.

Where changes are made across several devices at once, a per-device timer does not give a coherent rollback. The devices will revert at different times, and the intermediate state during reversion can be worse than either end state. Coordinating that requires an orchestrator that sequences the commits and treats the set as one transaction, and the timer on each device becomes a bound on how long the orchestrator may take rather than an independent safety net. The same coordination problem appears in the logging layer when several devices record the failure at different times with clocks that disagree, which is covered in correlating logs across devices.