Break-Glass Changes: Overrides That Are Auditable and Expire

Every configuration system meets the incident in which the declaration is wrong or too slow, and something must be changed on a running host before the declaration can be corrected. The two available responses are to change the host directly and repair the declaration afterwards, or to disable the configuration agent. The first loses the connection between declaration and reality; the second removes protection from the host for an indefinite period. A break-glass mechanism exists to make the emergency change a first-class, expiring, attributed declaration rather than an undocumented deviation.

The shape of an override

An override is data, not a bypass. It lives in the same hierarchy as the rest of the configuration, at a rank high enough to win, and it is scoped to the narrowest target that solves the problem: one host, one resource, one attribute.

The minimal override carries five fields, and each one exists because its absence has a specific failure.

The target: the resource identity the override applies to, expressed as the declaration would express it. Not “the nginx config” but the resource name, so that the override is attached to the same vertex in the dependency graph and does not create a second, parallel declaration of the same file.

The replacement value: the attribute and its value, not a whole-file replacement unless the whole file is the resource. Narrowing to an attribute keeps the rest of the declaration in force, so the override cannot silently revert an unrelated fix that lands while it is active.

The owner: an identity, so that the question “who changed this” has an answer that does not depend on reconstructing a shell history. Attribution is also the mechanism that makes the override socially expensive without making it technically hard.

The expiry: an absolute time, not a duration. A duration restarts whenever the override is re-applied, and an override that is re-rendered on every apply never expires. An absolute deadline means the override ends at a moment regardless of how often the loop runs.

The reason, in one line, for a human to read when the override is found active and nobody remembers why.

Why expiry is the load-bearing field

The value of the mechanism is entirely in the deadline. Without one, an override is a permanent change with better metadata, and the configuration converges to a state that is a mixture of declarations and the remnants of incidents.

With a deadline, the override has a lifecycle that the loop can act on without human intervention. Before expiry, the override participates normally: rendered, applied, and visible in the change report with its owner and reason. At expiry, the override stops applying, and the declared value takes over on the next pass. The transition is a change like any other, so it appears in the change report and, if the resource is a service configuration, triggers its notification. Overrides that target values which were moved outside the artifact have to be delivered through whatever mechanism supplies those values, and the deadline must be enforced there too, or the override becomes permanent in the one place no plan reviews.

The expiry is also the moment at which the emergency is declared over. If the declaration has been fixed by then, expiry is a no-op: the declared value equals the override, and the pass reports nothing. If it has not, expiry reverts the host to the value the declaration still says, which may reproduce the incident. That is the intended behaviour and the reason the deadline is chosen by the person making the change: it has to be long enough to cover the repair, and short enough that forgetting produces a visible failure rather than a silent one.

Expiry also has an unpleasant property worth understanding before deploying it. An override that is being relied upon and expires during an unrelated busy period produces a failure at a moment nobody chose. The mitigation is not to extend deadlines indefinitely but to make the deadline visible: a scheduled report of active overrides and their remaining time, so that the last hour is not the first time anyone looks.

Enforcement, detection, and the loop’s three settings

The configuration loop has three possible attitudes toward a host whose observed state differs from its declaration while an override is active, and they are not interchangeable.

Enforce. The loop reverts any difference, including one made by the operator during the incident. This is the default and it is correct for the vast majority of resources. It also means that a manual change during an incident survives only until the next pass, which may be seconds away. An operator who does not know the pass interval will conclude the change did not work.

Detect-only. The loop inspects and reports but does not apply. In this mode drift accumulates and nothing reverts it, so the mode is appropriate for a bounded window and for resources whose correct state is genuinely uncertain. The bound is what matters: a host left in detect-only mode is outside the loop’s protection, and the exit from the mode should be a condition (the override expiry, or a successful apply) rather than a manual step.

Quarantine. The loop skips the host entirely: no inspection, no apply. This is the most common accidental state, because it is what happens when the agent is stopped or its network path is broken, and it is the state from which hosts rarely return without an inventory audit. The important difference from detect-only is that quarantine produces no data at all, so a host in quarantine is invisible to drift detection and cannot be distinguished from a healthy host by any report that is derived from the loop.

Emergency overrides as described here use enforce mode with the override inside the declaration, which keeps the host under the loop while the deviation is recorded. That is the property that separates the mechanism from simply stopping the agent: the host is still converging, and what it is converging to is a declared value with an owner and an end date.

Preventing the override from becoming the configuration

Overrides accumulate. Three mechanisms keep the count bounded, and all three are necessary because each one alone is defeated by ordinary operational pressure.

The first is a cap per target. A resource with more than a handful of active overrides is not being patched, it is being rewritten through the override channel, and the declaration should be fixed. The cap is enforced by the tool, so adding the sixth override fails and the operator is directed to the declaration.

The second is a stock of open override tickets that grows monotonically until someone pays it down. New overrides are cheap; expired overrides that were never reconciled with the declaration are the debt. A declaration that was overridden for two weeks and never updated is now wrong, and the next apply will implement the wrong value. The paydown is mechanical: for each expired override, compare the override value with the declared value and either update the declaration or record that the declared value was correct.

The third is the identity of the approver. An override whose owner is the same identity that can approve it is not a control; it is a log line with a signature. Separation matters most for overrides that touch authentication, network policy, or the configuration system itself, because those are the overrides that turn an emergency into a durable change in the security posture.

What the model cannot cover

Two classes of emergency fall outside it, and one failure mode mimics an override that is not working.

The first is an emergency in which the configuration system itself is unavailable: the server that serves the catalogue is down, the network path to it is broken, or the credentials it uses have expired. No override can be delivered through a system that cannot be reached, and the fallback is necessarily local: the host’s agent applies its last cached catalogue, or an operator changes the host directly with the expectation that the loop will revert the change when it returns. The design question is what the agent does with a cached catalogue that it can no longer update: applying it keeps the host convergent to a stale intent, and not applying it leaves the host static. Either is defensible if stated; the failure is the ambiguity.

A distinct failure sits beside it and looks similar from the host’s point of view. The declaration is correct, the override is present, and the resource is still reported as changed on every pass. The usual cause is that the value reaches the resource through a different precedence layer than the one that was edited, which is the resolution question covered in how a configuration value resolves. The override did not fail to apply; it applied and was never the value the resource consumed.

The second is an emergency whose fix is a change the declaration cannot express at all, such as a manual failover that requires the primary’s identity to change in several places at once. The override mechanism expresses a value, not a procedure, and a multi-resource atomic change is a procedure. This is the case in which the loop should be paused deliberately, with the same attribution and deadline as an override, and the pause recorded where the rest of the audit trail is.