Deadline Propagation and the Shared Request Time Budget

A request passes through an edge proxy, an API service, and a storage service. Each hop sets a one-second timeout, each chosen by a different team, each defensible in isolation. The worst case end to end is three seconds. The client that started the request gave up after one second, closed the connection, and moved on. Two separate wastes follow from that arrangement: the client stopped waiting while two seconds of downstream work continued, and that work eventually completed into a socket that nobody was reading.

The problem is not the value of any individual timeout. It is that a duration has no memory. A one-second limit applied at each hop bounds each hop, and bounds nothing else.

A duration restarts; an instant does not

A timeout is a duration measured from the moment a call begins. A deadline is a point in time. The two are interchangeable only when there is exactly one call in the chain, and every real request crosses several.

Once a caller computes deadline = now + budget and passes the deadline itself, the arithmetic of the remaining budget is fixed for the whole chain: each hop subtracts the time already spent and forwards the same instant. Nothing is re-derived from a fresh local constant, so no hop can silently extend the total. This is the entire mechanism, and it is the reason deadline propagation is worth the plumbing it costs.

The distinction also determines what a failure means. A per-hop timeout expiring says that one hop was slow. A propagated deadline expiring says that the request as a whole has run out of time, which is the fact the caller actually needs, and which the failing hop can report without knowing anything about the caller’s budget.

What travels between services

An absolute instant on the wire requires synchronized clocks. Clock sources drift and occasionally step, and a deadline computed on one machine and interpreted on another inherits that disagreement directly. A deadline that is already in the past on arrival causes a request to fail instantly for no reason visible in either service’s own metrics.

gRPC avoids the problem by encoding a relative timeout rather than an instant. The grpc-timeout header carries a positive integer with a unit suffix from the set H, M, S, m, u, n, and the receiving side starts its own clock when it parses the header. The sender computes that duration from its own remaining budget at send time. The result is immune to clock skew, and it loses exactly the time spent in flight plus the time the request sat in the transport queue before the header was read. Under normal conditions that loss is a fraction of a millisecond. Under overload it is not, because the queue in front of the header parser is precisely where a busy server accumulates delay, and every hop that loses time this way is invisible to the budget arithmetic.

HTTP has no equivalent standard field. RFC 9110 defines no deadline header, so an HTTP-based system either defines one of its own and teaches every intermediary to forward it, or accepts that the chain is bounded only by the timeouts of its proxies. A custom header has to survive the proxy chain, which usually means explicit configuration: many proxies drop unrecognized headers or rewrite the field set.

Deriving the budget at each hop

The receiving service does not need the caller’s total budget. It needs two numbers: the deadline it inherited, and a floor below which issuing further work is pointless. If the remaining time is smaller than the floor, the correct action is to fail immediately and cheaply rather than start work that cannot finish.

const floor = 20 * time.Millisecond

// child shortens the inherited deadline so the caller's own error path
// still has time to run after the downstream call gives up.
func child(parent context.Context, reserve time.Duration) (context.Context, context.CancelFunc) {
	deadline, ok := parent.Deadline()
	if !ok {
		return context.WithCancel(parent)
	}
	if time.Until(deadline) < floor+reserve {
		// already out of budget: an expired deadline fails before any call is made
		return context.WithDeadline(parent, time.Now())
	}
	return context.WithDeadline(parent, deadline.Add(-reserve))
}

The reserve is the part most implementations omit. When a downstream call is allowed to consume the entire remaining budget, its failure arrives at the same instant the caller’s own deadline expires, so the caller cannot send a meaningful error response and the client sees a timeout rather than the actual cause. Carving out a fixed slice, on the order of tens of milliseconds or a fixed fraction of the budget, converts a large share of deadline expiries into informative failures.

Go’s context package is a natural carrier because deadlines are inherited by construction: a child context created with WithDeadline on a parent that already has an earlier deadline is bounded by the earlier one. Languages without an equivalent ambient mechanism usually carry the deadline in an explicit request-scoped struct, and the risk there is a code path that constructs a fresh scope and drops the field.

Work that outlives its caller

Propagating a deadline is necessary and not sufficient. A deadline only does anything if the work checks it. Three failure patterns recur.

The first is the blocking call that cannot be interrupted. A handler waiting on a mutex, a non-context-aware database driver, or a synchronous file read will not observe cancellation until the operation returns on its own schedule. The deadline fires, the client disconnects, and the goroutine or thread remains occupied for the full duration of the underlying operation. From the outside the request is cancelled; from the inside the resource is still held.

The second is work that is queued rather than running. A request that arrives with two hundred milliseconds left and waits four hundred milliseconds for a worker slot has already failed, but nothing checks that until it starts. Checking the deadline at dequeue and discarding the item before executing it costs a comparison; skipping that check costs a full execution whose result is thrown away. The queue dynamics behind that delay are covered in Queue Time, Service Time, and What a Timeout Measures.

The third is the side effect that lands after cancellation. A cancelled request can still have written a row, published a message, or charged a card, because cancellation is a signal rather than a rollback. The work around those operations needs its own bounded, non-cancellable scope with a short explicit deadline, plus idempotency so that a retry after cancellation does not duplicate the effect. Cancellation Propagation and the Work That Outlives Its Caller treats the mechanics of that scope in detail.

Choosing the number

An end-to-end budget belongs to the caller and follows from its service level objective, not from the comfort of any internal hop. A common shape is a fixed total, with allocations that shrink as the chain deepens: the edge holds the whole budget, the first internal service is allowed most of it, and the leaf gets the remainder minus the reserves. Allocations are upper bounds, and the propagated deadline enforces them without each hop needing to know the allocation.

The budget also has to cover every attempt, not every attempt individually. A retry policy of three attempts with a two-hundred-millisecond timeout each does not fit a four-hundred-millisecond budget; the second attempt starts with nothing left. Retry Budgets and the Load Multiplication They Prevent takes the interaction between attempts and budget as its subject.

A single fixed timeout is also a statement about the failure rate it is willing to accept. If healthy responses complete within one second at the 99.9th percentile, a one-second timeout fails roughly one in a thousand healthy requests by construction, before any dependency misbehaves. Setting the value at a high percentile of the measured distribution makes the acceptable false-failure rate an explicit choice rather than an accident.

Where the arithmetic breaks

Deadlines assume every hop participates. A third-party dependency that ignores the header and applies its own thirty-second limit breaks the chain at that point, and no amount of care upstream fixes it; the only remedies are a client-side bound and a circuit breaker on that dependency.

Long-lived streams are another mismatch. A deadline on a connection that is meant to stay open for hours is not a bound on any individual operation. Idle timeouts and per-message deadlines measure the things that actually matter there, and a request deadline applied to the whole stream converts a healthy long connection into a guaranteed failure at the deadline instant.

Finally, a deadline that expires is not the same as work that is abandoned. The gap between the two is measured in whatever the slowest non-interruptible operation in the path costs, and it is the reason a service can report low latency to clients while its own CPU stays saturated on requests whose answers were never delivered.