Deadline Propagation Across Service Hops

A caller that gives up after two seconds does not stop the work it started. The request continues in the callee, holding a connection, a thread, and whatever database transaction it opened, and its result is written to a socket that no one is reading. If the callee has its own two-second timeout and calls three further services, the abandoned work can run for six seconds after the caller has already returned an error.

This is the failure that per-hop timeouts create when they are configured in isolation. Each service enforces its own limit, no service knows the caller’s limit, and the total work admitted into the system is set by the sum of the limits rather than by the one that matters.

A timeout that only the caller honours

The purpose of a timeout is to bound the resources a request can consume, not merely to free the caller. A timeout that frees the caller while leaving the callee working has bounded only half the problem, and under load the unbounded half is the half that causes the outage.

The mechanisms that follow from this are all about carrying the caller’s limit into the callee. The caller knows when it must have an answer. The callee must know the same instant, so that its own work, its own outbound calls, and its own queue waits all fit inside it.

Absolute deadlines and the remaining budget

Two forms of limit are transmitted. An absolute instant, such as “do not answer after 12:00:00.500”, is unambiguous but depends on synchronised clocks. A duration, such as “1700 ms remaining”, avoids clock assumptions and requires each hop to subtract the time it has already spent.

Both forms exist in practice. The duration form is safer because NTP adjustments, virtual machine clock drift, and container migrations can move a wall clock by seconds, and a deadline compared against a skewed clock either expires early or never. Monotonic clocks, such as CLOCK_MONOTONIC on Linux, are unaffected by wall-clock corrections and are what a runtime’s timer facility uses internally. A duration measured from a monotonic start and compared against a remote instant is a mixture that only works if the clocks agree, which is why deadline implementations transmit a remaining duration and start counting when it arrives.

gRPC does exactly this. The client sends a grpc-timeout header whose value is a duration with a unit suffix, and each server computes the remaining budget for its own work, including time spent waiting for a connection and any retries it performs. When the budget is exhausted the call fails with DEADLINE_EXCEEDED, and the client cancels the underlying stream. The deadline applies to the whole call rather than to individual message reads, so a slow stream of responses does not extend it.

HTTP has no equivalent standard header. Services that need one either propagate a documented custom header carrying milliseconds remaining, or derive it from the trace context and the entry timestamp. Whatever the mechanism, the arithmetic is the same at every hop: take the budget on arrival, subtract elapsed time, reserve a margin for the response path, and pass the remainder.

func handle(w http.ResponseWriter, r *http.Request) {
    // Budget arrives as milliseconds of remaining time, not as an absolute clock value.
    budget := 800 * time.Millisecond
    if v := r.Header.Get("X-Request-Budget-Ms"); v != "" {
        if ms, err := strconv.Atoi(v); err == nil {
            budget = time.Duration(ms) * time.Millisecond
        }
    }
    if budget <= 50*time.Millisecond {
        // Too little time to do useful work; fail fast instead of starting it.
        http.Error(w, "deadline exceeded", http.StatusGatewayTimeout)
        return
    }

    // Leave 50 ms for serialising the response and closing the connection.
    ctx, cancel := context.WithTimeout(r.Context(), budget-50*time.Millisecond)
    defer cancel()

    if err := callInventory(ctx, r.Header.Get("X-Request-Budget-Ms")); err != nil {
        http.Error(w, err.Error(), http.StatusGatewayTimeout)
        return
    }
}

Cancellation is not automatic

Propagating a deadline is only half the mechanism. The other half is that the callee notices.

At the transport level, a cancelled gRPC call sends an HTTP/2 RST_STREAM, and a cancelled HTTP/1.1 request closes the connection. Both are visible to the server’s runtime, which converts them into whatever cancellation signal the language exposes: a cancelled context in Go, an aborted signal in a browser fetch, an interrupted thread in Java only if the code checks for it. None of these stop execution by themselves in a language without preemptive interruption. A handler that ignores the signal runs to completion, and the only resource saved is the socket write.

The gap that matters is the blocking calls inside a handler. A database query started before cancellation continues on the server until the driver is told to stop, and drivers vary in whether they expose that. Go’s database/sql cancels a query when its context is cancelled, which is a property of the driver’s support for context. In JDBC the equivalent is a statement timeout or an explicit Statement.cancel() from another thread, and neither happens automatically when an HTTP request is aborted. A bulk operation over a result set needs periodic cancellation checks, or it will keep consuming CPU and holding locks long after the client has gone.

The observable symptom of missing cancellation is a service that shows low load on its own request-rate graph while its database shows high utilisation. Requests that were abandoned upstream are still doing work, and they do not appear in the completed-request metrics that capacity planning uses.

Budget arithmetic across a chain

Dividing a budget is a design decision with a visible effect on error rates. The total must cover the client’s network hop, every server’s processing, the slowest downstream call, and the return path.

A 800 ms end-to-end target with a single backend hop might allocate 50 ms for the client-to-edge leg, 650 ms of backend work, and 100 ms of slack. A backend that then calls three services in series cannot give each of them 650 ms; it must decide how the 650 ms splits, and the split is a statement about which dependency is allowed to be slow.

Serial composition adds the limits, parallel composition takes the maximum. A handler that calls three services in parallel and waits for all of them can allocate the full remaining budget to each, because the wall-clock cost is the slowest. A handler that calls them in series must allocate a fraction, and the last call in the chain inherits whatever is left. This is another reason to avoid long serial chains: the budget at the end of a five-hop chain is a small fraction of the budget at the start, and the last hop fails most often for reasons entirely outside its control. The same chain length also divides the composed availability, which is the arithmetic in Availability Multiplication in Synchronous Service Graphs.

Timeouts also have to be larger than the transport’s own retransmission behaviour. TCP’s initial retransmission timeout is one second by default per RFC 6298, so a timeout set below that fires before the network has had a chance to recover from a single lost packet. The result is a class of failures that appear as server errors while the server never received a request, and that disappear from any investigation focused on server logs.

Retries consume budget twice. Each attempt takes time, and the total across attempts must still fit the deadline; a client that retries three times with a two-second per-attempt timeout against a two-second deadline has no time for the second attempt unless the first fails fast. Retry budgets, expressed as a fraction of total requests rather than a per-request count, keep the additional load bounded during an incident. The Site Reliability Engineering material on overload describes both the retry amplification and the budget approach. Whether a retry is safe at all is a property of the operation rather than of the deadline, and it is the subject of Idempotency Keys and Duplicate Suppression at Service Boundaries.

Where deadline propagation breaks down

The mechanism assumes that every intermediary participates. A request that passes through a proxy that recreates headers, a queue that stores messages for minutes, or a batch job that submits work on a schedule has left the synchronous chain, and a deadline expressed in milliseconds means nothing on the other side.

Asynchronous work needs a different signal: an expiry time recorded with the message, checked when the message is consumed, so that work whose result is no longer useful is discarded rather than performed. A nightly reconciliation job that picks up messages from the previous day is not helped by a 500 ms budget, and applying one simply makes every message fail.

The second gap is queueing inside the callee. A deadline that is checked when the request arrives does not prevent work from being queued behind other requests and starting after the deadline has passed. Admission control that compares the deadline against the current queue depth, and rejects requests that cannot possibly finish, is what prevents a service from spending its capacity on work whose results will be discarded.

The third gap is the proxy idle timeout, which is often shorter than the application’s deadline and is configured in a different file by a different team. When the proxy gives up first, the client sees a 504 with no body while the server continues and eventually logs a successful response. Aligning the layers takes deliberate comparison of every timeout in the path, from the client library to the load balancer to the server to the database statement timeout, and the shortest one defines the real behaviour.