Reliability Block Diagrams and Markov Availability Models

An availability target of 99.99 percent sounds like a small step above 99.9 percent, and in downtime it is a factor of ten: about 4.3 minutes per month against about 43. The figure is arithmetic, not aspiration, and it can be computed for a proposed architecture from a block diagram before a single machine exists. The diagram also shows the part that usually gets the answer wrong — the redundancy that shares a failure domain with the thing it is supposed to protect.

Downtime stated in the same units as the work

Availability is the fraction of time a system delivers its service, over a stated window, against a stated definition of service. The definition is load-bearing: a checkout service that accepts orders but cannot take payment is down by one definition and degraded by another, and the two produce different numbers from the same event.

Converting a target into time makes the target falsifiable:

Target	Downtime per 30-day month	Per year
99%	about 7 h 12 min	about 3.65 days
99.9%	about 43 min	about 8.8 h
99.95%	about 22 min	about 4.4 h
99.99%	about 4 min 19 s	about 52 min
99.999%	about 26 s	about 5 min

The arithmetic is (1 - A) x window, with a month taken as 30 days = 43,200 minutes and a year as 525,600 minutes. The useful comparison is against the change process: a deployment that requires a two-minute restart per instance, rolled across twenty instances without overlap, spends forty minutes of downtime against a 99.99 percent monthly budget of about four. A target that cannot survive the ordinary operation of the system is not a target; it is a statement that the system needs a different deployment mechanism.

Two quantities get confused here. Reliability is the probability of operating without failure over an interval; availability is the fraction of time the service is usable, which counts repair. A component with a long mean time between failures and a slow repair can have the same availability as one that fails often and recovers instantly, and the two designs feel entirely different to operate. For a service whose availability target is expressed as uptime, A = MTBF / (MTBF + MTTR), so repair time is as important as failure rate.

Series blocks: small numbers compound

In a reliability block diagram, blocks in series represent components that must all work for the system to work. Their availabilities multiply:

A_series = A_1 x A_2 x ... x A_n

Five components at 99.9 percent give 0.999^5 ≈ 0.995, which is 99.5 percent, or about 44 hours of downtime a year rather than 8.8. Ten such components give 99.0 percent. The compounding is the reason a system assembled from individually respectable services misses its target: every synchronous dependency on the critical path is a multiplier, and a chain of ten is a chain of ten.

Two properties follow directly. Adding a dependency costs availability in proportion to the dependency’s own unreliability, so a component at 99.99 percent costs a tenth as much budget as one at 99.9 percent. And making the critical path shorter is the only structural fix; tuning an individual component helps, but the product is dominated by the worst term.

The blocks must be assigned to the same definition of service. A dependency that is not on the critical path — an analytics call that is asynchronous and lossy — is not a series block, and putting it in the diagram understates availability. A dependency that is nominally optional but whose absence blocks a user flow is a series block, and leaving it out overstates it.

Redundancy and the shared failure domain

Parallel blocks represent redundancy: the system works if at least one block works, so

A_parallel = 1 - (1 - A_1)(1 - A_2) ... (1 - A_n)

Two components at 99 percent give 99.99 percent, which is why replication looks like a bargain. The formula has one assumption: the failures are independent. Almost every real outage in a redundant system violates it.

def series(*availabilities):
    result = 1.0
    for a in availabilities:
        result *= a
    return result

def parallel(*availabilities):
    down = 1.0
    for a in availabilities:
        down *= (1.0 - a)        # all replicas must fail for the pair to fail
    return 1.0 - down

critical_path = series(0.999, parallel(0.99, 0.99), 0.9995, 0.9999)
print(f"{critical_path:.5f}")          # about 0.99830 for these inputs

Shared failure domains are the reason the parallel result is optimistic. Two replicas on the same hypervisor fail together when the host fails. Two replicas in the same rack share a power feed and a switch. Two replicas in the same region share the control plane that manages them, the DNS that resolves them, the certificate that authenticates them, and the deployment system that will restart them both from the same bad configuration. In each case the correct diagram has the replicas in parallel inside a series block representing the shared dependency, and the shared block is frequently less available than the replicas it contains.

The usual accounting error is to count replicas as independent and leave out the shared layer entirely. A pair of replicas at 99.9 percent behind a load balancer at 99.95 percent and a control plane at 99.9 percent yields, as a series composition, about 99.85 percent, a tenth of a point better than the 99.75 percent a single replica would give. The redundancy bought three nines of component reliability and the shared layers spent most of them.

Correlation also comes from the change process, which block diagrams rarely represent. A bad configuration pushed to every instance in a rolling deployment is a failure that no component availability figure captures, because the components were working. The mitigation is in the deployment design — canary, automatic rollback, blast-radius limits — and the diagram should include it as a block if the target depends on it.

Markov states when repair and degradation matter

Block diagrams assume each component is up or down, independently, and that the system’s state is determined by the combination. When repair competes with failure, or when a degraded state has different failure rates, a continuous-time Markov chain expresses the same system more accurately and at a cost in size.

The two-state chain is the smallest useful model. Healthy is state 0, failed is 1, failure rate λ and repair rate μ. The steady-state availability is μ / (λ + μ), which is the same as MTBF / (MTBF + MTTR) since MTBF = 1/λ and MTTR = 1/μ for exponential durations. Concretely, a component failing once a year on average with a one-hour repair gives A ≈ 0.999886, which is about one hour of downtime a year.

Three states make the model earn its keep. Consider a pair of replicas where one failure degrades the service, the second failure takes it down, and repair of the last failure is slower because a replacement has to be provisioned. States are shown as (2, 1, 0) healthy replicas, with transitions 2 → 1 at rate 2λ, 1 → 2 at μ_fast, 1 → 0 at λ, and 0 → 1 at μ_slow. Solving the balance equations gives the steady-state probability of state 0, and the answer is sensitive to μ_slow in a way that the parallel block formula cannot express at all.

The general method is to write the generator matrix Q over states and solve πQ = 0 with Σπ = 1. For small chains this is a linear solve; software for it exists in every reliability toolkit and the implementation for a handful of states is a short Gaussian elimination.

The cost is state-space growth. n independent components with two states each give 2^n states; twenty components give about a million, and thirty give a billion. The practical approach is hierarchical: use a block diagram for the independent majority, collapse each subtree to a single availability figure, and use a Markov model only for the small set of correlated or repairable parts. The decomposition is exact only if the collapsed subtree’s behaviour is memoryless with respect to the rest, which is an approximation — a reasonable one when the subtree recovers much faster than the rest of the system.

Detection and failover are part of the model

Every redundancy scheme has a window between the failure and the restoration of service, and the window is often longer than the failure itself would suggest. Three components determine its length.

Detection: a health check with a 10-second interval and a threshold of three consecutive failures takes up to 30 seconds to notice, during which requests go to a failed instance. Faster detection costs false positives, which cause failovers that were not needed.

Switchover: promoting a replica, re-resolving DNS, or electing a leader takes time proportional to the mechanism. DNS-based failover is bounded by record time-to-live, which for a 60-second TTL means a minute of partial traffic to the failed target even after the change is published.

Recovery of state: an instance that comes back empty needs to catch up, and during catch-up it may serve stale data or refuse traffic. A cache that must refill after a restart is unavailable or slow for the duration of the refill.

For a target expressed in minutes per month, a 30-second detection plus a 60-second switchover plus a 5-minute warm-up consumes the monthly budget of 99.99 percent in a single event. That is the calculation that makes a failover drill worth running: it measures the window, and the window is the term the block diagram omits.

Partitioning assumptions and honesty about them

A model built from vendor availability figures inherits vendor assumptions about what counts as unavailability, and those definitions differ. Some count scheduled maintenance, some do not; some count regional failures, some only instance-level. Composing numbers from different sources in one series product produces a figure with no defined meaning.

The defensible practice is to state the definition alongside the number, use figures from the same source where possible, and treat the result as an order-of-magnitude bound rather than a prediction. A model that says a design is around 99.9 percent and cannot reach 99.99 without removing a series dependency has done its job. A model that says 99.9937 percent has claimed a precision that the inputs never supported.

There is one more use for the diagram that is worth more than the number: it forces the failure domain question to be asked explicitly for each pair of replicas. Where the answer is “they share the rack”, the diagram is documenting a decision to accept that risk, which is exactly what the scenario set should record. Where the answer is “nobody knows”, the diagram has found the gap before an incident does, and the evaluation depth question answers itself.