Dependency Objectives: Propagating Budgets Across a Call Graph

A service with its own objective and a set of dependencies has a problem the objective does not solve: the objective is a statement about outcomes at its own boundary, and the failures that prevent those outcomes mostly originate somewhere else. The dependency’s objective and the caller’s objective are both true, they are measured over different populations, and there is no arithmetic that turns one into the other. Treating a dependency’s objective as an input to the caller’s budget — “upstream promises 99.95%, so 0.03% remains for the caller’s own failures” — is a plausible construction with a specific error at its centre.

Independence is the assumption that fails

If a request completes only when every one of its dependencies responds successfully, and each dependency fails independently with probability $q_i$, the caller’s success probability is $\prod_i (1 - q_i)$. At three dependencies each at 99.9%, the product is 99.7%: the caller cannot exceed 99.7% no matter how perfect its own code is. This is the source of the familiar observation that a service cannot have a target tighter than the product of its dependencies’ targets plus its own failure modes.

The independence assumption is what makes the product wrong in both directions. Dependencies share failure domains: a network partition, a control-plane outage, a DNS failure, or a saturated host affects several dependencies at once. Under a shared cause, the failures are perfectly correlated and the caller’s availability is the minimum of the dependencies, not the product — considerably worse than the independence model. Conversely, the caller may have invented redundancy: retries, fallbacks, cached responses, degraded modes. Under successful fallback, dependency failure may cause no request failure at all, and the correct model is not a product but a function of the caller’s own fallback logic.

The practical consequence is that dependency composition is not arithmetic over targets. It is a property of the call graph plus the code’s handling of each edge, and the only measurement that captures both is the caller’s own indicator, which already contains the effects. The caller’s objective is the primary quantity; dependency objectives are instruments for diagnosis, not inputs to a calculation.

What a dependency objective is actually for

Given that, the dependency’s published target has three legitimate uses. The first is as a contract: it states what the dependency intends to provide, and it is the reference against which the caller decides whether a design that requires 99.99% of that dependency is realistic. The second is as an attribution tool: when the caller’s burn rises, comparing its ratio against each dependency’s ratio over the same period localizes the source faster than log search. The third is as an early warning: a dependency trending toward its budget limit is a signal that the caller’s own objective is at risk before the caller’s numbers move, which is the point at which mitigation can still be cheap.

The illegitimate use, which is common because it is convenient, is as a fault-attribution mechanism in the financial sense: charging the caller’s consumed budget to the dependency that caused it and settling accounts. This requires knowing which dependency caused each failure, and a failure at the caller’s boundary does not carry that information. An error response from the caller after a timeout does not say whether the database was slow, the network dropped the packet, or the caller’s own thread pool was exhausted. Establishing causation requires per-request trace data with per-span status, and even then the causal edge for a timeout is genuinely ambiguous: a slow dependency and an insufficient caller deadline produce the same observation.

Fan-out turns a per-call target into an end-to-end impossibility

The composition problem is at its sharpest with fan-out. Consider a service that assembles a page from $k$ independent shards, requiring all $k$ to respond. If each shard honours a 99.9% target, the probability that all respond is $0.999^k$. At $k = 100$, that is about 90.5%. At $k = 1{,}000$, about 36.8%. The same arithmetic, expressed for latency under a threshold $T$: if each shard exceeds $T$ with probability $\epsilon$, the probability that at least one of $k$ does is $1 - (1-\epsilon)^k$, which for $k = 100$ and $\epsilon = 0.001$ is about 9.5%. A shard-level 99.9% tail budget becomes a 90% whole-response tail at a hundred-way fan-out.

Nothing about this is avoidable by setting a tighter per-shard target; it is a property of the conjunction. There are three exits and all of them change the system rather than the objective.

Reduce $k$ per request. Partition so that a single request touches fewer shards. This trades parallelism for per-shard capacity, since each shard must now handle more data or more load.

Make the conjunction partial. Serve the response from a subset, tolerating missing shards. A page that renders with 95 of 100 results is not a failure, and the objective at the caller’s boundary should reflect that. The measurement requirement is then a graded one, which the ratio form does not express directly; the usual approach is to define the criterion as “at least $m$ of $k$ parts returned” and score that boolean per request.

Add a hedge or a backup. Send the request twice after a delay and use the first response. The tail-at-scale literature documents hedging and tied requests as the standard tools; both convert the tail probability from $\epsilon$ to roughly $\epsilon^2$ for the hedged case, at the cost of duplicated load — which is bounded by the hedge rate, but which also interacts with the system’s own overload behaviour in ways the per-call target does not describe.

The choice among them is a design decision with a capacity cost, and the objective at the caller’s boundary is what makes the cost visible. A team that writes the end-to-end target first and then derives the per-shard target will discover that the derived number is unattainable and will either widen the objective or abandon it.

Deadlines are the propagation mechanism for latency budgets

For latency specifically, the mechanism that connects objectives across a call graph is the deadline, not the target. A caller that has 300 milliseconds to respond and needs four sequential calls can allocate roughly 75 milliseconds to each, with the remaining slack held as a buffer. Passing the remaining time as a deadline in the request means that a dependency which is running slow fails fast rather than consuming the caller’s entire budget and returning a timeout at the same instant the caller gives up.

Three properties matter. The allocation must leave slack, because the caller’s own processing and the network are not free, and a perfectly partitioned deadline turns every hop’s variance into caller failure. The deadline must decrease along the path, monotonically, or a downstream service will continue working on a request whose caller has already abandoned it, consuming capacity for an outcome that is already a failure. And the dependency’s latency objective must be stated at a threshold at or below the deadline it is given, since a dependency that meets a 300-millisecond objective while the caller’s allocated deadline is 75 milliseconds meets an irrelevant objective.

Clock skew makes the wall-clock deadline hostile in practice: two hosts disagreeing by tens of milliseconds can make a deadline appear already expired on arrival. Implementations therefore usually propagate a duration rather than an absolute timestamp, or rely on clock synchronization with a known bound. Either way, the budget is consumed by the deadline mechanism regardless of the target, and the target is only the accounting view of it.

Aggregation across the graph is not the sum of the parts

At fleet scale, the objective that matters to the end user spans several services, and no single service’s objective describes it. A request path of six services each at 99.95% has an end-to-end availability of about 99.7% if the failures are independent, and the user’s experience is what the composition produces, not what any one service reports.

Two approaches are used to measure this directly. The first is a black-box check at the edge, which is an objective over synthetic or real user traffic that traverses the path; it is the only measurement that reflects the composition, and its cost is that it does not decompose. The second is trace-derived metrics: counting spans with error status at each service and joining them by trace identifier to compute path-level success. Traces make the composition observable but are sampled, so the counts carry the sampling uncertainty treated in low-volume objectives and counting qualifiers, amplified at the path level because a path requires all its spans to be recorded to be assessable.

The consistency requirement that follows is that any objective quoted for a user-facing path be measured on that path, at the edge, over a stated population, using the denominator rules set out in the SLI ratio: numerator, denominator, and measurement point. Objectives quoted per service are useful only when the reader knows they are partial; a path-level target assembled by multiplying per-service figures and presented as the user’s experience is the most common defect in a dependency review, since the result corresponds to no measurement taken at any boundary.

Where the contracts have to go

The dependency contract that survives contact with operation is a short document per edge with four fields: the dependency’s target and window, the failure modes it explicitly does not cover (for instance, an overload rejection under a load-shedding regime), the caller’s fallback if the dependency fails, and the maximum extra latency the caller will tolerate before failing. The last field is the one that turns the other three into a testable design constraint, since it determines whether the caller can absorb a dependency’s tail or is exposed to it.

Retry behaviour belongs in the same document, and it is where dependency objectives are most often violated from the caller’s side. A caller that retries three times on any failure converts a dependency’s 0.1% error rate into a 0.3% request amplification during an incident, applied precisely when the dependency is least able to absorb it. The caller’s own objective is then met while the dependency’s budget is destroyed, which is the clearest illustration of why the two quantities cannot be composed arithmetically: they are not measuring the same events, and one of them can be spent to protect the other.