The Degraded Window: Rebuild Rate, Exposure, and the Second Failure

A redundancy scheme’s failure tolerance is usually quoted as a static number: three copies tolerate two failures, a 10-of-14 code tolerates four. The number describes the configuration, not the system in operation. The system spends part of its life below its configured tolerance, and the length of that time, measured against the failure rate, determines what the tolerance actually buys.

Exposure as an integral

Define the exposure of a stripe as the number of additional failures it can absorb right now. It is m, the parity count, when everything is healthy. It drops to m-1 the moment a fragment is lost, stays there while the rebuild runs, returns to m when the rebuild completes. If a second failure arrives during the window, exposure drops to m-2, and a third failure before either is rebuilt loses the stripe.

The probability of losing a stripe over a horizon is the probability of enough failures arriving inside the windows created by earlier failures. For independent failures at per-device rate λ, the dominant term for a scheme with tolerance m is a sequence of m+1 failures in which each of the last m arrives within the rebuild window opened by its predecessor. The loss rate therefore carries m factors of the window duration and m+1 factors of the failure rate: it scales approximately as $\lambda^{m+1} T^{m}$ up to a constant, where T is the time to restore one failure’s redundancy. The window is not a second-order correction; it enters with the same exponent as the failure rate that follows it.

That is the whole argument for fast repair. Halving the rebuild time divides the loss rate of a two-failure-tolerant scheme by four, and of a one-failure-tolerant scheme by two. Nothing else in the system changes: the same disks, the same code, the same copy count, a different exposure integral.

Why rebuilds are slow

The rebuild duration is set by the slowest of several resources, and the slowest is usually not the one the operator expects.

The first bound is the read volume. Rebuilding one lost replica under three-way replication reads one replica’s worth of data; under a 10-of-14 code it reads ten fragments per rebuilt fragment, which is ten times the data on the lost device. A device holding 10 TB therefore generates 100 TB of reads during its rebuild, and the network and the source disks must carry that. If the cluster’s spare bandwidth is a small fraction of its total, the rebuild duration is the read volume divided by the spare bandwidth, and it is measured in hours or days rather than minutes.

The second bound is the per-source limit. The k fragments are read from k distinct devices, so each source serves one fragment per stripe, and a source that is also serving foreground traffic has a limited ability to serve repair reads. Rate limiting is applied per source rather than globally for this reason, and the effective repair rate is the minimum over sources.

The third bound is the target’s write rate. The rebuilt fragments must be written somewhere, and the target device’s write bandwidth is finite. On a cluster where the target is also a serving device, the write competes with reads.

The fourth bound is the metadata and coordination cost. A rebuild enumerates the objects that were on the lost device, which requires a scan of the placement metadata; for a large cluster that scan is not instantaneous, and the rebuild is effectively idle while it completes. This is why rebuilds often show a slow start followed by a steady rate: the enumeration phase precedes the data movement.

The self-limiting feedback loop

A rebuild makes the cluster busier, and a busier cluster serves foreground requests more slowly, which in some systems causes retries, which adds more load. The loop is not usually catastrophic, but it does mean that the repair rate achieved under load is lower than the rate measured on an idle cluster, and the degraded window is therefore longer than a naive calculation suggests.

The same loop operates through the failure rate itself. A device under sustained heavy read load runs hotter, and elevated temperature is associated with higher failure rates in fleet studies. A rebuild that reads a device at its maximum for days is therefore applying exactly the conditions that make the next failure more likely, at exactly the moment when the next failure is most damaging. The correlation is not a design flaw to be fixed; it is a constraint that argues for capping repair rate below the device’s maximum and accepting a longer window in exchange for not accelerating the second failure.

m = 2 (tolerance), T = rebuild duration, lambda = per-device failure rate

P(a second failure lands inside this window)  ~  lambda * T
P(stripe lost from this window)               ~  (lambda * T) * lambda

so the loss probability scales linearly with T for a one-step window,
and the exponent of lambda is one more than the tolerance consumed

The arithmetic is deliberately crude; the point is the proportionality, which holds under any reasonable model. The window is a first-class term in the durability expression and not a rounding error.

What actually gets rebuilt

The naive rebuild replaces every lost fragment with a new fragment at full redundancy. Two refinements change the exposure profile substantially.

The first is prioritisation. Not all data is equally valuable, and not all stripes are equally exposed. A stripe that has already lost one fragment and is waiting for repair is more exposed than a stripe that is healthy; a stripe whose surviving fragments sit on devices that are themselves showing errors is more exposed still. Rebuilding the most exposed stripes first reduces the probability of loss during the window even when the total rebuild takes the same time. The information needed is a per-stripe exposure count, which is cheap to maintain and is the reason many stores track a “degraded” flag per stripe rather than only a per-device failure state.

The second is partial rebuild. If a lost device held fragments for many objects, the rebuild can restore the fragments for the most exposed objects first and the rest later, which is the same prioritisation applied at object granularity. Some systems go further and rebuild only enough parity to restore the tolerance, deferring the reconstruction of data fragments that are cheap to reconstruct later; this trades a longer period of read amplification for a shorter period of exposure.

The second failure is not independent

The exposure model above assumes failures arrive independently at rate λ. The assumption is weakest exactly where it matters, because the second failure is often caused by the same event as the first: a power event takes out a rack, a firmware bug crashes a batch of drives, a maintenance action reboots a host that was already serving as a rebuild source.

The practical consequence is that the effective λ during the window is higher than the steady-state λ, sometimes by a large factor, and the window’s contribution to loss probability is correspondingly larger. This is why failure domains and repair scheduling are coupled: a rebuild that reads from sources in the same rack as the failed device is reading from devices that share the failed device’s power and cooling, so the sources are more likely to fail next than a random device would be. Choosing repair sources by failure domain, not just by availability, is the mitigation, and it costs read locality, since the closest sources are often the ones in the same rack.

Setting the rate

The repair rate is a policy with two failure directions. Too slow and the window dominates durability. Too fast and the repair traffic degrades foreground latency, and the degraded devices are pushed harder.

A workable rule is to size the rate so that the expected time to restore full redundancy for a single device failure is small relative to the mean time between failures at the cluster’s population size. If the cluster has D devices with per-device rate λ, failures arrive at roughly Dλ per unit time, and the cluster is in a degraded state whenever a rebuild is in flight. For the degraded fraction to stay small, the rebuild duration must be well under $1/(D\lambda)$; for a cluster of a thousand devices with an annual per-device rate of two percent, that is a failure roughly every 18 days, and a rebuild of a few hours keeps the degraded fraction in the low single digits of a percent.

The rule is a sanity check rather than an optimisation. The real constraint is that the repair path must be able to keep up with the failure arrival rate indefinitely, because a cluster whose repair capacity is below its failure arrival rate has a queue that grows without bound, and the degraded fraction approaches one. A cluster in that state is not degraded occasionally; it is permanently degraded, and its stated redundancy is a description of a state it never reaches. The check is therefore the first thing to compute when sizing a cluster, and it is a property of the code’s read amplification as much as of the network.

The interaction with correlated placement is the remaining piece. A window that is short is worth little if the failures that arrive inside it are correlated, because a single correlated event can consume the remaining tolerance in one step. The exposure integral and the failure-domain constraint have to be satisfied together; either one alone gives a durability estimate that the deployment does not deliver.