Failure Domains and the Arithmetic Behind Replication Factor

Three copies of a block sounds like a number chosen by intuition. It is actually the output of a calculation, and the calculation has a premise that is easy to state and hard to satisfy: the copies must be able to fail separately. Every durability claim a distributed store makes is conditional on that premise, and every placement decision in the system exists to keep the premise true.

What one extra copy buys

Start with a single device. A drive that fails at a rate of roughly one to two percent per year is within the range documented in large fleet studies, so a single-copy object has a durability of about 0.98 to 0.99 over a one-year horizon. That is not a storage product; it is a coin flip weighted slightly in the user’s favour.

Add a second copy on a different device that fails independently. The object is lost only if both fail inside the repair window, and the probability of that is the product of the individual probabilities, near $0.02 \times 0.02 = 4 \times 10^{-4}$ per year, assuming repairs keep pace. A third copy multiplies again, to about $8 \times 10^{-6}$. The exponent is the copy count, and the base is the per-device annual failure rate. This is the whole reason replication factors are small integer: each increment of one buys roughly two orders of magnitude of durability, and the returns diminish only because the base is already small.

The multiplication is legitimate only when the events are independent. Independence is a property of the placement, not of the disks. Two drives in the same chassis share a power supply and a fan; two chassis in the same rack share a top-of-rack switch and a circuit breaker; two racks in the same room share cooling and a fire suppression system; two rooms in the same region share a power grid and a fibre path. Each of those shared components converts independent device failures into a single correlated event that takes out every copy at once. The arithmetic above is not wrong, it is conditional, and the condition is what a failure domain hierarchy encodes.

Domains as nested sets

A failure domain is a set of components that can fail together. Domains nest: a disk is inside a host, a host inside a rack, a rack inside a room, a room inside a region. The rule that placement must enforce is that no two copies of the same object live in the same domain at whatever level the operator has declared meaningful. With three copies and a three-level hierarchy, the strict reading is one copy per rack, with the racks themselves in different rooms if the durability target requires it.

The hierarchy is a statement about correlation, not about geography. A domain boundary is meaningful only if the failure it contains is actually contained. Two racks on separate power feeds are separate domains for power loss and one domain for a network partition if they hang off the same aggregation switch. This is why real systems do not model a single tree of domains but a set of constraints: a placement policy asks for copies on distinct hosts, distinct racks, and distinct power zones, and the constraints overlap rather than nest cleanly.

# Illustrative placement check: copies must not share any declared domain.
def violates(copies, domains):
    """copies: list of dicts mapping domain-kind -> domain-id.
       domains: kinds that must be distinct, e.g. ['host', 'rack', 'power']."""
    for kind in domains:
        seen = [c[kind] for c in copies]
        if len(set(seen)) != len(seen):
            return True
    return False

The check is trivial; the difficulty is that the set of copies changes. A rebuild moves a copy, a node decommission removes one, and a rebalance reshuffles many. Every mutation is an opportunity to produce a placement that satisfies the constraint syntactically while violating its intent, for example by putting the rebuilt copy on the one host in the rack that is already serving another copy of a neighbouring stripe.

Durability is not availability

The two words get used interchangeably and they measure different things. Durability is the probability that the bytes still exist; availability is the probability that a request succeeds now. Replication helps both, but not in the same way and not in the same direction.

Three copies give a very high probability that the data survives, because loss requires simultaneous failure of all three. They give a much lower probability that a write can be acknowledged at full redundancy, because a single node down means the write is either waiting, or being accepted at reduced redundancy, or being routed to a replacement. A store configured for three copies spends a noticeable fraction of its life with some objects at two, and a smaller fraction at one. The durability calculation above already accounts for that if the repair rate is fast enough; if repair takes hours and failures arrive independently at a per-device rate, the effective copy count is lower than the configured one for the duration of every repair.

The rebuild window is therefore part of the durability specification, not an operational detail. So is the answer to what happens when the third copy is unavailable: a store that refuses writes preserves durability and sacrifices availability; a store that accepts writes at two copies preserves availability and lowers durability for the objects it touched. Neither is wrong. A store that does not say which it does is underspecified.

The cost axis and where the money goes

The space cost of replication is linear and blunt: three copies consume three times the raw capacity, and the usable fraction is one third. Erasure coding trades that for computation, and a k-of-n code stores n/k times the data, so a 10-of-14 code costs 1.4x instead of 3x. The saving is real and the price is paid in three places: encoding and decoding CPU, a read path that must fetch k fragments instead of one, and a repair path that must fetch k fragments to rebuild a single lost one.

That last item is the one that surprises. Under three-way replication, rebuilding a lost copy means reading one copy and writing one copy, a 1x amplification of the object. Under a 10-of-14 code, rebuilding one lost fragment means reading ten fragments and writing one, a 10x amplification. Since repair traffic is what the cluster spends most of its background bandwidth on, the choice between replication and coding is less a durability decision than a decision about which resource is scarce: capacity or interconnect. A cluster with cheap disks and an oversubscribed network prefers replication; one with expensive media and spare bandwidth prefers coding.

Correlated loss is the dominant term

If failures were independent, three copies would be enough for any plausible durability target, and the design space would collapse to a single question about cost. In practice the tail of the loss distribution is driven by correlation: a rack loses power, a switch firmware upgrade reboots a top-of-rack, a cooling failure raises inlet temperatures until a whole row of drives shuts down.

This is the argument behind placement schemes that bound the number of copies that share a failure set rather than merely counting copies. Random placement of three replicas across a large cluster produces a distribution over which nodes hold each object; the tail of that distribution contains objects whose three copies happen to sit in the same rack, and those objects are lost by a single event. Making placement deterministic and constraint-aware removes the tail, at the cost of a placement function that must be recomputed whenever the cluster’s shape changes, which is the subject of consistent hashing and membership change.

There is a second-order effect that no amount of copy counting removes. Data loss events are not uniform over time; they cluster around maintenance. A human replacing a failed drive in a rack already at reduced redundancy is a correlated failure in the same sense as a power event, and the window in which it matters is the same window.

Reading a redundancy claim

A durability figure such as eleven nines is a model output, and the model has visible inputs: the per-device failure rate, the repair rate, the independence assumptions, and the size of the population. Eleven nines over a million objects is a different claim from eleven nines over a thousand, because the expected number of losses scales with population while the per-object figure does not.

The useful response to such a claim is not to check the exponent but to ask what failure set was assumed independent. If the answer is “individual disks” and the placement puts three copies in one rack, the exponent is fiction. If the answer is “racks, with power and network separately constrained”, the number is a statement about a specific physical layout and holds only for that layout. Durability numbers are properties of a deployment, not of a storage engine.

The complementary mechanism, erasure coding, changes the arithmetic rather than the premise. A k-of-n code with n = k + m survives any m failures, so the exponent becomes a combinatorial count of how many distinct failure sets of size m+1 exist, and the space cost drops from n copies to n/k times the data size. The premise is unchanged: the m+1 failures must still be independent, and Reed-Solomon coding buys space at the price of a repair path that reads more than one block per rebuilt block.