Availability Multiplication in Synchronous Service Graphs

A request handler that calls three services before it can answer has an availability problem that none of the three services has on its own. If each callee is available 99.9% of the time, the handler cannot exceed 99.7%, because a request fails when any dependency it needs fails. Nothing about the handler’s own code changes that. The number is a property of the call graph, not of the process.

That arithmetic is the first thing to internalise about distributed architecture. It also explains a common surprise: a service whose own error rate is negligible appears in dashboards as the least reliable component in the system, because it sits at the top of a chain of dependencies that each carry their own risk.

Requests as a dependency chain

Availability is the fraction of a measurement window during which a component can serve a correct answer. A 30-day month contains 43,200 minutes, so 99.9% allows 43.2 minutes of unavailability, 99.95% allows 21.6, and 99.99% allows 4.32. These figures set the budget that any composition has to fit inside.

For a request that needs every one of n dependencies, and assuming failures are independent, the composed availability is the product:

A_total = A_1 * A_2 * ... * A_n

Five dependencies at 99.9% produce 99.5%. Ten produce 99.0%, which is over seven hours of monthly downtime for a system whose parts each promise under an hour. Fifty produce roughly 95%, a number no customer-facing service survives. The product falls faster than intuition suggests because each new edge takes a fresh slice of the remaining budget.

Redundancy composes the other way. For k replicas where any single one can answer, availability is one minus the product of the failure probabilities. Three replicas at 99% give 1 − 0.01³, or 99.9999%. That asymmetry is the entire reason replication exists, and it is also why redundant reads and serial writes have such different reliability profiles: a write that must reach a quorum behaves like a series chain over the quorum members, not like a parallel one.

The graph that matters is the one for a specific response mode. A product page served from a local cache has one edge, to the cache. The same page with personalisation and live stock has four. Both are “the product page service”, and quoting one availability number for it is meaningless unless the split between modes is known.

from math import prod

def series(*availabilities):
    return prod(availabilities)

def parallel(*availabilities):
    return 1.0 - prod(1.0 - a for a in availabilities)

page = series(0.999, 0.999, 0.999, 0.995)   # four required callees
cached = 0.9995                             # one local read, no network hop
print(f"live path {page:.3%}, cache path {cached:.3%}")

What the product hides

The multiplication rule is a lower bound on downtime, and it is wrong in both directions once real failure behaviour enters.

Failures correlate. Components sharing a rack, a zone, a network device, a DNS resolver, a certificate, or a release process fail together. In a series chain, correlation makes the product pessimistic: if two dependencies always fail at the same moment, the chain loses that interval once, not twice. The composed number looks better than the arithmetic predicts while the event itself is worse, because a single outage removes the whole chain at once. Availability measured as total minutes cannot distinguish one 43-minute outage from 43,000 one-second failures, and the second is far more damaging to a client with retries.

Shared dependencies get counted more than once. If two callees both require the same database, the chain contains that database twice, and the product understates true availability. Modellers usually flatten such graphs to distinct failure domains before multiplying, which is also the reason a shared database destroys the value of splitting a system into services in the first place.

Partial correctness does not fit the model. A service that returns stale data, or an empty result set instead of an error, is available by the binary definition and wrong in practice. Error budgets expressed as request success rates therefore need a separate correctness signal, or the number quietly improves as quality degrades.

Turning hard edges into optional ones

The useful architectural move is not raising a dependency’s availability from 99.9% to 99.99%, which is expensive and often impossible, but deleting the edge or making it optional. Four techniques recur.

Caching and local replicas replace a remote call with a local read. The edge remains, but its failure probability becomes the cache’s, and a cache hit does not depend on the origin being up. The cost is staleness, which must be bounded and stated: a price cache with a 60-second refresh is a different product decision from one with a one-hour refresh.

Graceful degradation removes the dependency from the response path when it is unavailable. Recommendation panels, avatars, view counters, and “customers also bought” are candidates; payment authorisation is not. Degradation is only real if the fallback is exercised, because a fallback that has never run is a second bug waiting for the first outage.

Denormalisation moves data to the caller at write time. The caller now holds a copy of a field it does not own, so the edge is replaced by a staleness contract and an update path. This is the mechanism behind most cross-service read models, covered in Read Models Across Service Boundaries.

Aggregation at the edge collapses several round trips into one call from the client. It does not reduce the number of backend edges; it reduces the number of places where a partial failure has to be handled, and it shortens the latency chain.

Each of these trades a hard dependency for a soft one plus a consistency cost. Naming that cost explicitly is what separates a design decision from an accident.

Fan-out, retries, and the shape of the tail

Fan-out multiplies risk. A request that scatters to 100 shards and needs all of them has 90.5% availability if each shard is 99.9%, and the arithmetic gets worse with every shard added. Systems that fan out widely usually accept partial results, returning whatever answered within a deadline and marking the rest as missing, which changes the composition from “all must succeed” to “enough must succeed”.

Fan-out also changes latency. When a request waits for the slowest of many parallel responses, the expected latency is driven by the tail of the individual distributions, not the mean. A dependency whose 99th percentile is ten times its median will dominate the fan-out’s latency even though its median is unremarkable. Hedged requests, where a second copy is sent after the first has been outstanding past the median, trade extra load for tail reduction; the cost is a few percent more requests at normal times and a much larger increase during overload.

Retries are the mechanism that converts a transient dependency failure into a sustained one. Three attempts per request triple the offered load at the moment the dependency is least able to absorb it, and retries stacked at several layers multiply rather than add: a client retrying three times against a gateway that retries three times against a service that retries three times produces 27 attempts per original request. Retry budgets, where retries are capped as a fraction of total requests rather than per request, keep the amplification bounded. The Site Reliability Engineering chapter on cascading failures treats this as the central mechanism by which a small overload becomes a total outage.

Timeouts do not improve availability; they change its shape. A dependency that hangs consumes a connection, a thread, and a queue slot for as long as the caller waits, so an unbounded wait turns a slow dependency into a caller-wide failure. A deadline converts that into a fast, countable error and frees the resource. How the remaining budget is divided across hops is the subject of Deadline Propagation Across Service Hops.

What to do with the number

The output of this arithmetic is not a single availability figure to put in a design document. It is a ranked list of edges, each carrying the failure probability of the callee, the share of traffic that traverses it, and a decision: remove it, replicate it, make it optional, or accept the exposure. Edges with high traffic share and low callee availability come first, and edges that sit on a write path deserve more scrutiny than read-path edges because their failure loses data rather than delaying it.

Two checks keep the analysis honest. First, flatten the graph to distinct failure domains before multiplying, so shared databases and shared control planes are counted once. Second, compare the resulting number with the error budget the product actually promises; a composed availability below the promise means the architecture has to change, and no amount of per-service hardening closes a gap that the multiplication rule created.