Propagating Context Across Queues, Threads, and Proxies

The single hardest part of tracing is not recording spans. It is making sure that when a unit of work leaves the place where it started, the identifier of the span that caused it goes along. Every implementation that looks simple at first is simple because it assumes the work stays on one thread inside one process, and that assumption is false for essentially every system that needs tracing in the first place.

The implicit current-span variable and where it stops working

The convenient interface is a thread-local or goroutine-local notion of “the current span”. Starting a span pushes it onto an implicit stack; finishing it pops. Code deep inside a call tree can then create a child span without any of the intermediate functions taking a context parameter. This works because the runtime guarantees that the executing thread is the one that entered the function.

It breaks in four common places.

A worker pool hands the task to a different thread. The thread-local variable belongs to the thread, not the task, so the worker sees whatever span the thread was last running, which may be an unrelated request or nothing at all. Worse, if the pool reuses threads, a stale span from a previous request can be picked up and the child span is attached to the wrong parent, producing a trace that looks plausible and is wrong.

An asynchronous callback resumes after the initiating function has returned. The span stack has already been unwound, so the callback runs with an empty context even though it is causally downstream of the request.

A message is published to a broker and consumed seconds later, possibly by a different service on a different host. There is no thread, no stack, and no shared memory: the only channel is the message payload and its metadata.

A request crosses a proxy or a load balancer that terminates the connection and opens a new one. Header names and values are copied by configuration, and any header not explicitly forwarded is lost.

The response to all four is the same in principle: context must be an explicit value that is passed along with the work, and every boundary must have a defined carrier for it. The differences are all in what the carrier is.

Process boundaries: headers as the carrier

For synchronous request-response traffic, the carrier is the request header set, and the format is fixed by the trace context specification described in the traceparent header format. The client serialises its current span identifier into the outgoing traceparent value, and the server parses it and creates its own span with that identifier as parent. Nothing else is needed: no shared memory, no session, no affinity.

The failure mode here is almost always an intermediary rather than the endpoints. A reverse proxy configured to forward a fixed list of headers will drop traceparent unless it is added, and the resulting trace shows a gap exactly at the hop where the header vanished. Frameworks that provide their own tracing integration usually inject and extract automatically for their own HTTP client and server abstractions, but a hand-written socket call or a raw client library bypasses that integration and silently loses the header. The symptom is a childless span: a client span with no matching server span at the other end, which a reader can detect by looking for client spans whose duration roughly matches a server span that is not in the trace.

Header case handling deserves a mention because it produces confusing failures. HTTP header names are case-insensitive, so a proxy that rewrites traceparent to Traceparent is still conformant at the HTTP level, and a receiver that looks headers up case-sensitively will fail to find it. Most frameworks normalise header names on read; those that do not require the extraction code to normalise first.

Thread and task boundaries: the carrier is the task

Inside a process, the carrier is whatever object represents the unit of deferred work. In a language with structured concurrency the answer is the language’s own context type, threaded through every function that might spawn work. In a language without one, the answer is a wrapper that captures the context at submission time and restores it at execution time.

The critical property is capture at submission, not restoration at execution. Consider a pool where the submitting thread holds span A and the worker thread happens to hold span B from an unrelated request. If the task reads the current span when it starts, it gets B. If it captured A at submission, it gets A regardless of what the worker was doing. The difference is invisible in a single-threaded test and produces cross-linked traces under load.

// Capture at submission: the wrapper closes over the context that was
// active when the task was handed to the pool, not when it runs.
final class TracingRunnable implements Runnable {
    private final Context captured = Context.current();
    private final Runnable delegate;

    TracingRunnable(Runnable delegate) { this.delegate = delegate; }

    @Override public void run() {
        Context prev = Context.current();
        try (Scope s = captured.makeCurrent()) {
            delegate.run();
        } finally {
            // Restore the worker's previous context so the next task
            // does not inherit this one.
            prev.makeCurrent();
        }
    }
}

The finally block is not optional. A worker thread that finishes a task without restoring the previous context leaves the completed task’s span installed, and the next task on that thread will parent its spans to a span that has already ended. A span whose parent has a later end time than its own is one of the few structural errors a trace reader can detect reliably.

For callback-based APIs, the same rule applies with the callback as the task. The context must be captured where the callback is registered, not where it is invoked. Libraries that accept a callback and later invoke it from an internal event loop are the usual offenders, because the registration site and the invocation site are in different execution contexts by construction.

Queue boundaries: the carrier is the message

Asynchronous messaging has no request to attach headers to, so the context travels in message metadata. The two common brokers both provide a place for it: AMQP has a headers table on the message properties, and Kafka has record headers, a byte-array key-value list separate from the payload. Neither interprets the values; both store and forward them.

The producer injects traceparent into the metadata and creates a producer span. The consumer extracts it, creates a consumer span with the producer span as parent, and continues. This preserves the causal chain across an arbitrarily long delay, which is the point: a trace can span minutes or hours of queue latency, and the resulting duration is the real end-to-end latency including waiting.

Two complications arise. The first is that a batch consumer receives many messages in one poll and processes them in one loop. The natural span structure is one consumer span per message, but the loop itself does real work, and the work is shared across all the messages. Recording one span per message with the same parent set is legitimate, and links are the mechanism for the many-to-one case discussed in span anatomy: the processing span links to all the producer spans rather than parenting to one of them arbitrarily.

The second is that the trace now spans a trust boundary in time. A message may be consumed by a service that did not exist when the producer was deployed, and the producer’s header format may have changed. Version negotiation in the header is what makes this survivable, since an old consumer can ignore extension fields it does not understand.

There is a subtler issue with retries and dead-letter queues. When a message is redelivered, the consumer creates a new span for the new attempt, but the parent is still the original producer span. That is correct causally, and it also means the retry appears as a sibling of the first attempt rather than as its child. Some systems instead re-inject a modified context so that the retry chains from the failed attempt; both are defensible, and the choice should be recorded in the trace by naming the attempt number as an attribute so a reader can tell them apart.

Proxies, gateways, and the header allowlist

An intermediary that forwards traffic must forward the tracing headers, and the practical problem is that the list of headers to forward is usually a security allowlist rather than a passthrough. A gateway that forwards only Authorization, Content-Type, and Accept will drop traceparent and tracestate silently.

When an intermediary is configured to forward them, a second question appears: should it also create its own span? For an L7 proxy that parses requests and applies policy, a server span and a client span at the proxy are meaningful and show the time spent inside the proxy. For a pure TCP load balancer that cannot see headers, there is no span to create, and the trace simply skips that hop, which is a correct representation of the fact that the balancer does not participate.

A gateway that terminates TLS and re-originates the request is a case where header forwarding alone is insufficient. The trace identifier survives, but the parent identifier must be updated to the gateway’s own span, otherwise the upstream service parents to a span two hops back and the gateway’s work is invisible. This is just the normal client-server rule applied at the proxy: extract the incoming context, start a server span, start a client span for the outgoing call, inject the client span’s identifier.

Detecting a broken chain

Propagation failures are silent by default, so the useful practice is to make them visible. Three checks catch most of them.

The first is a structural check on the trace: every client span should have a corresponding server span with the same operation, and a server span whose parent is missing from the trace is a propagation failure or a sampling inconsistency. The second is to record the propagation mechanism itself as an attribute on the span, such as whether the context came from a header, from a message, or was generated locally, so a query can count spans that started a new trace unexpectedly. The third is to test the boundaries directly, by running a request through the real proxy configuration and asserting that the trace identifier observed at the far end matches the one generated at the near end. One request per boundary per deployment is sufficient for the assertion, and the cheap version is a synthetic probe with a known identifier, since it does not need to be sampled or stored.

The cost of getting this wrong is not a missing feature but a systematically misleading dataset: traces that appear complete while silently omitting entire services. That is a worse failure than no tracing at all, because it supports conclusions that the data does not.