Asymmetric Partitions and What a Set of Conflicting Suspicion Reports Means

Every detector described so far produces an opinion about a peer, and every such opinion is formed by observing one path. The moment the network stops being symmetric, those opinions stop agreeing, and the disagreement is not noise to be averaged away. It is the primary evidence about what has actually happened, and a system that discards it loses the ability to tell a dead machine from a broken wire.

The observation matrix

Model the cluster as a set of nodes and, for each ordered pair, whether the first can currently exchange messages with the second. This is not symmetric. A firewall rule can permit traffic in one direction only. A routing change can black-hole one direction while the reverse path stays up. A middlebox can drop one side’s packets while the other side’s packets are accepted. A link can be saturated in one direction and idle in the other. The matrix has n(n-1) entries and the interesting faults are the ones where it is not symmetric and not diagonal.

Each detector samples one row of that matrix: the row belonging to the observer, restricted to the peers it probes. A report of “peer p is dead” is really a report that the entry (observer, p) has been unresponsive for the threshold duration. It says nothing about the entry (p, observer), nothing about (q, p) for other observers q, and nothing about whether p is running.

The matrix framing makes the diagnosis problem precise. There are three hypotheses consistent with a row of failures:

• p is down. Then every column entry for p fails and every row entry from p is absent.
• The observer’s path to p is broken. Then the entry (observer, p) fails and other observers may still reach p.
• p is up but unable to respond, for example because it is paused or its outbound path is broken. Then (observer, p) fails while (p, observer) may also fail, and other observers see a mix.

A single observer cannot separate these. Two observers can separate the first from the second, provided their paths do not share the broken component. The full matrix can separate all three, and the pattern of agreement is what identifies the fault. A design that collapses the matrix into a single upward report, such as a hierarchy of monitors, gives up this distinction by construction.

Reading the pattern

The useful diagnostic is not the count of suspicions but their shape.

A column failure, where every observer reports the same peer unresponsive, is consistent with the peer being down and also with the peer’s own outbound path being broken, since both produce silence in every direction. These two are not distinguishable from outside the peer. They are, however, equivalent for most purposes: a peer that cannot send or receive is not usable, and whether its process is running does not change what the cluster should do.

A row failure, where one observer suspects many peers and the peers do not suspect it, is the signature of an isolated observer. Its own network is broken, or its own process is paused, and it is the one that is wrong. This is the case that a naive membership protocol gets backwards: the isolated node declares everyone else dead, and if its declarations are accepted, the cluster removes healthy nodes on the evidence of a broken observer. The protocol must weight an observer’s reports by whether that observer is itself reachable, which is circular unless the reachability evidence comes from independent sources.

A block pattern, where a group of nodes suspects another group and the suspicion is mutual and complete across the boundary, is a partition. Inside each group the traffic is fine, across the boundary nothing passes. The cluster is split, and the question becomes which side may act, which is a quorum question rather than a detection question. Quorum arithmetic decides that only one side can hold a majority, and the minority side must stop serving even though its members are healthy and can reach each other.

A partial block, where only some cross-boundary pairs fail, is the hardest case. It is consistent with an asymmetric routing fault, a one-way firewall change, or a middlebox that has lost state in one direction. No membership rule handles it cleanly, because each node’s view is locally correct and globally wrong.

Why an asymmetric link is not a partition

A partition is usually defined as a failure that prevents some pair of correct processes from communicating. That definition covers asymmetry: if a cannot send to b, then a and b cannot communicate, and the system is partitioned regardless of whether b can send to a. The distinction that matters is not whether the partition exists but whether the protocols above it can still make progress.

A consensus protocol that requires request-response in both directions treats an asymmetric link as a partition, because it needs a reply. A protocol that only requires one-way dissemination, such as a gossip channel, can make progress over an asymmetric link: b’s updates reach a, a’s do not reach b. That is exactly the situation in which a node receives news about itself that it cannot refute, and it is why refutation needs an outbound path that the accused node controls.

The practical consequence is that the fault model has to be stated at the level of the operation, not the level of the link. “The network may partition” is too coarse; “a message may fail to arrive and the sender may not learn that it failed” is the property the protocols actually have to survive.

Making diagnosis actionable

The first rule is to keep the observer’s identity attached to the observation. Merging suspicions into a boolean membership state throws away the matrix, and the matrix is the only thing that supports a diagnosis. A membership entry that says “peer p is dead” is less useful than one that says “three observers cannot reach p, one can, and p’s own reports arrive at that one”. The second form allows a system to conclude that p is reachable but degraded, which is often the correct action: stop routing new writes, keep reading.

The second rule is to distinguish reachability from responsiveness at the point of measurement. A probe that fails because no acknowledgement arrived is a statement about the path plus the peer. A probe that fails because the local send call returned an error is a statement about the local path. Recording which one happened costs a field and makes the row-versus-column distinction computable rather than inferred.

The third rule is that recovery must not depend on the disputed resource. A node that is cut off cannot prove it is alive to the cluster through the cut. It needs an out-of-band path, a different network plane, or a rule that admits it back after a bounded time without evidence. The first two are engineering; the third is a policy that trades safety for availability and must be chosen deliberately. Lamport’s parliament analogy is relevant here: a member that cannot attend is not part of the quorum, and its return is a new event rather than a continuation.

What the system may safely conclude

The safe conclusions are weaker than most systems assume, and stating them exactly is the useful part.

A node may conclude that it cannot currently exchange messages with a peer. That is a fact about the node’s own experience and is always true when observed.

A node may conclude that a peer is not serving it. That follows if the failure is sustained, and it is the conclusion that justifies routing around the peer. It does not justify removing the peer from a quorum, because removal is a global decision and the node has only local evidence.

A group may conclude that a peer is gone if every member of the group fails to reach it and the group’s members do not share a plausible common cause. The second clause is the one that is usually skipped. A rack-level switch failure satisfies the first clause and violates the second, and treating it as a peer failure is the detection storm described in correlated timeouts.

A group may conclude that it is partitioned from another group if the failure is mutual and complete across the boundary. Even then, the correct action is determined by the quorum rule, not by the detector, because both sides are internally healthy and only the configuration says which one may proceed.

Nothing in this list supports the conclusion that a peer’s process has crashed. That conclusion is never available from network evidence alone. It is a useful fiction that lets the system act, and the design should make the fiction cheap to reverse, which is why graded suspicion and quarantine rather than immediate removal recur throughout the mechanisms in this cluster. The detector’s contribution is to shorten the time during which the fiction is wrong, not to make it true.