Fencing and Quorum in Automated Failover

Failover is usually described as detecting a failure and promoting a replica. The detection is the visible part and the promotion is the satisfying part, but neither is where the design difficulty lies. The difficulty is that a failed node and a slow node are indistinguishable from the outside, and a node that has been declared failed may still be running, still holding its data, and still willing to accept writes.

If it does, the system has two primaries. Each accepts writes, each generates a divergent history, and by the time the partition heals there is no correct way to merge them: the same primary key has two rows, the same account has two balances, and one of the two sets of client acknowledgements has to be retroactively declared a lie.

The write that must not happen

Every failover protocol is built around a single ordering requirement. Before the new primary accepts its first write, the old primary must be unable to accept any. Not unlikely to, not about to stop, but unable.

The requirement is not satisfied by the old primary agreeing to stop. An agreement is a message, and messages are what the partition dropped. It is not satisfied by a timeout either: a timeout is evidence that a reply did not arrive, which is consistent with a dead node, a busy node, a slow network, and a node whose replies are being dropped by an asymmetric route. All of those look the same from the outside, and only one of them is safe to treat as death.

The mechanisms that satisfy the requirement fall into two families. One makes the old primary physically incapable of writing: cut its power, disable its network port, revoke its access to the storage. The other makes its writes recognizable as stale, so that whoever receives them can reject them. The first family is called fencing, and the second is called a fencing token, and the second only works if the resource being written checks the token.

Leases expire, processes do not

The common coordination primitive is a lease: the authority grants a node the right to act as primary for a bounded interval, and the node may act only while it believes the interval has not elapsed.

The safety argument depends on two assumptions that are easy to violate. The first is that the leaseholder’s clock does not run fast relative to the authority’s. If it does, the holder believes it still has time after the authority has given the lease to someone else. The second is that a process which stops executing does not resume with its belief intact. A virtual machine that is suspended by the hypervisor, a container that is frozen by the scheduler, or a process stopped by a debugger can all resume after an interval during which its lease expired, with no awareness that time passed. It wakes believing it is still primary.

This is not a theoretical concern for virtualized infrastructure, where pausing a VM to migrate it, snapshot it, or back it up is routine. The pause can last long enough for the lease to expire and a new primary to be promoted, and the resumed VM will happily continue writing.

The defences are a watchdog that fires when the process fails to check in, and a fencing token that makes the stale writes detectable. A watchdog is a hardware timer that resets the machine if software does not service it within the interval; the software cannot be paused and still service the timer, so a pause becomes a reset. The token is a number issued with each lease, increasing monotonically; every write to storage carries the token, and the storage rejects any write whose token is lower than the highest it has seen. A stale primary’s writes then fail at the storage layer, which is the only place that can adjudicate.

Three layers of fence

Fencing can be applied at whichever layer the failing node can be reached through, and each layer has a different failure characteristic.

Power fencing switches the node off, through a power distribution unit, a baseboard management controller, or a hypervisor API. It is unambiguous and slow: the command has to be delivered, the machine has to lose power, and the storage has to observe that the node’s I/O has stopped. A node that is wedged in a way that keeps its I/O in flight can continue writing for as long as the controller’s cache holds the data.

Network fencing disables the switch port or installs a filter that drops the node’s traffic. It stops the node from talking to clients and peers, and it does not stop the node from talking to storage if storage is on a separate network. Applied alone, it can produce a node that is isolated from the cluster but still writing to the shared volume, which is the worst of the available states.

Storage fencing revokes the node’s access to the volume, either by removing a LUN mapping, by revoking a reservation, or by having the storage array reject I/O from that initiator. This is the layer that matters most when the data is on shared storage, because it directly enforces the property the other layers only approximate.

Self-fencing is the fourth option and the one that scales: a node that cannot renew its lease kills itself, on the theory that a dead node cannot write. It requires the node to have both the ability and the determination to do so, which means a watchdog that is independent of the operating system’s scheduler and a reset path that does not depend on the network that just failed.

Cluster managers generally require a fencing device to be configured before they will manage a resource that can run in two places. The requirement is not bureaucracy. A cluster that promotes a replica without fencing has automated the promotion and left the split brain to chance.

Quorum is a cheaper authority

Fencing needs a device that can reach the node, and that device is another component to operate, authenticate, and keep available. Quorum avoids the device by making the decision depend on a majority.

With an odd number of nodes, any two partitions cannot both contain a majority. A node that cannot reach a majority cannot confirm that it is the primary, and a well-built system refuses to serve writes in that state. The partition that holds the majority continues; the partition that does not stops. There is no scenario in which both sides write, because the arithmetic forbids it.

The cost is availability, and it is paid in exactly the situation operators find most alarming: a two-node cluster where one node fails has no majority, so the survivor cannot serve. The usual fix is a witness, a lightweight third participant that does not store data but votes, restoring the arithmetic without doubling the cost.

Quorum protects the coordination layer, not the data layer. A database primary that is not part of the quorum system can still be writing while the quorum says it should not, unless the database’s promotion path consults the quorum. Distributed consensus stores such as etcd make this explicit: losing quorum means the store stops accepting writes, which is a safe failure that looks like an outage.

Two-node clusters without a witness are the recurring source of split brain in practice, because the natural failover rule, “if the peer is unreachable, take over”, is exactly the rule that produces two primaries under an asymmetric partition. A rule that requires a majority, or a fencing confirmation, is slower and sometimes refuses to act, and both of those are the point.

Why the safe configuration looks like an outage

Every safety mechanism here converts a possible divergence into a possible refusal. The cluster that cannot fence does not fail over. The node that cannot reach quorum does not serve. The storage that sees a stale token rejects the write. Each of these is a service outage that a less careful configuration would have avoided, in the sense that it would have kept answering requests while quietly creating two versions of the truth.

The design question is therefore not how to eliminate the refusals but how to make them short and visible. Fencing devices should be reachable from both nodes and from the network that survives the failure, not from the network that just failed. Quorum should include a witness that is not colocated with either data node. The refusal should raise an alert that names the missing authority, because an operator who cannot tell “the peer is down” from “the fence failed” will disable the fence.

Recovering from a split brain that has already happened is a different problem, and it ends with one side’s history being discarded. On PostgreSQL, a former primary that diverged is not eligible to rejoin as a standby without being rebuilt or rewound, because its log contains transactions the new timeline never saw. Deciding which side to discard is a data-loss decision, made under pressure, with incomplete information, and it is the outcome the fencing design exists to prevent.

The ordering constraints that a failover imposes on everything above the database are covered in recovery ordering for multi-service estates. The loss window that remains even when fencing works is set by the replication configuration described in log shipping and the recovery point objective.