Clock Skew and the Arithmetic of Span Durations

Every span carries two timestamps taken from the clock of the machine that wrote it. No tracing protocol synchronises those clocks. Yet duration queries work, latency percentiles are meaningful, and the waterfall view is usually correct. The reconciliation is that a trace uses time in two distinct ways, and only one of them is vulnerable to skew. Confusing them produces either false confidence or unnecessary panic.

Two uses of time, one of which is safe

The first use is the duration of a single span: end minus start. Both readings come from the same clock, on the same machine, microseconds apart. Any constant offset in that clock cancels in the subtraction. What remains is the clock’s rate error, which for a disciplined oscillator is on the order of parts per million, and the granularity of the clock source, which is typically nanoseconds on modern hardware but may be coarser. Over a span of ten milliseconds, a rate error of 50 parts per million contributes half a microsecond. The duration is sound.

The second use is the relationship between spans written on different machines: the parent starts before the child, the child ends before the parent, the gap between a client span’s end and the server span’s start is the network round trip. These comparisons subtract readings from different clocks, and any offset between those clocks appears directly in the result. A machine whose clock is 40 milliseconds behind its peer makes every operation it initiates look 40 milliseconds shorter on the wire and 40 milliseconds longer inside the callee, and the sum still equals the true elapsed time, which is why the error is invisible in aggregate totals and obvious in per-hop breakdowns.

The practical rule that follows: trust durations, distrust absolute timestamps, and treat any quantity computed by subtracting timestamps from different hosts as an estimate whose error bound is the clock offset between them. The durations themselves are the fields that a span record is built around, and they are the ones that survive skew intact.

Where the offset comes from

Machines synchronise time with NTP or a similar protocol, and the achievable accuracy depends on the network path and the local oscillator. On a local network with a stable reference, offsets in the low hundreds of microseconds are typical. Across the public internet, offsets of a few milliseconds are common and tens of milliseconds occur when the path is asymmetric or congested. Virtual machines add their own complication: the virtual clock may be steered by the hypervisor, may pause when the guest is descheduled, and may jump when the host migrates.

The distinction that matters for tracing is between wall clock and monotonic clock. The wall clock is the one NTP adjusts, and it can move backwards when a correction is applied. The monotonic clock never moves backwards and is unaffected by NTP steps, but it has an arbitrary epoch and no meaning outside the machine. A tracing library that reads the wall clock for both endpoints of a span can produce a negative duration if the clock steps backwards between the two readings, which happens during a large correction. Reading the monotonic clock for the duration and the wall clock only for the start timestamp avoids the negative case entirely, at the cost of storing two values. Many libraries do exactly this and report the duration from the monotonic source.

A clock step also damages the trace structure in a way a rate error does not. A rate error makes spans consistently early or late by a small amount; a step of several seconds, applied between the writing of a parent span and a child span, puts the child in a completely different part of the timeline. Skew that accumulates slowly is a nuisance; a step is a discontinuity that no correction can fix without knowing when it happened.

The impossible waterfall

The visible symptom of skew is a waterfall in which a child span extends beyond its parent’s interval. A server span that starts before its client span, or ends after it, is not a bug in the instrumentation. It is the arithmetic of two clocks disagreeing.

host A clock:  +0 ms
host B clock:  +12 ms (B is 12 ms ahead)

A: client span  [====================]
B:   server span       [====================]
                       ^ appears to start 8 ms after the client ends

Reading the diagram naively suggests the network took a negative time, or that the server responded before being asked. A trace viewer that clamps such spans to fit inside the parent hides the anomaly and produces a picture that is tidy and false. A viewer that shows the raw timestamps exposes the anomaly but invites the reader to draw conclusions about the network that the data cannot support.

The correct reading is that the uncertainty in the cross-host comparison is the offset, and any interval smaller than the offset is not resolvable. For a client-server pair, the network round trip computed as client duration minus server duration has an error of the offset between the two hosts. If that round trip comes out as 0.4 milliseconds on a pair of hosts whose clocks may differ by 2 milliseconds, the measurement is dominated by the error, and the honest conclusion is that the round trip is somewhere below the offset bound.

What correction can and cannot do

There are two families of correction.

The first is to fix the clocks. This is the only approach that preserves absolute timestamps, and it is a prerequisite for anything that correlates traces with logs or metrics from different machines. Disciplined timekeeping, a consistent time source for all hosts, and monitoring of the offset itself are the operational answer. The tracing system can help by recording the local clock offset as a span attribute at write time, if the host exposes it, so a later analysis can bound the error instead of guessing.

The second is to correct the trace after collection, using the constraints the tree imposes. The idea, described in the literature on clock skew in distributed tracing, is that a parent must enclose its children, so any violation gives an inequality on the unknown offsets. With enough parent-child pairs across a set of hosts, the offsets can be estimated by finding values that minimise the total violation. This is a linear program in the number of hosts, not the number of spans, and it can be solved offline.

The limits are what make this a partial answer. The constraints only bound the offsets; they do not determine them. A set of offsets that satisfies every enclosure constraint is a valid solution, and there may be a family of them, particularly when the trace graph is shallow or when all the spans on one host are adjacent in time. The estimate also depends on the assumption that no span is genuinely misparented, since a single wrong parent link becomes a false constraint and pulls the solution. And correction can only be applied to traces already collected; it cannot help a live decision, such as deciding whether a request is slow enough to sample, which must use the timestamps as recorded.

A third option, common in practice, is to record enough information to reason about the error without pretending to remove it: the span’s duration from the monotonic clock, the wall-clock start, and an attribute for the host’s estimated offset. A reader can then choose whether to compare across hosts at all.

Asymmetric paths and the one-way delay problem

Even with perfect clocks, a client-server split cannot separate network time from server time without an assumption. The client knows the total round trip; the server knows its own processing time; the difference is the sum of the outbound and inbound network delays. Splitting that sum requires assuming symmetry, and paths are frequently asymmetric: a request may be larger than the response, or the routing may differ in each direction.

This is the same problem NTP solves with its four-timestamp exchange, and it has the same answer: the estimate is bounded by the asymmetry, and the bound is not observable from the timestamps alone. For tracing, the consequence is that a “network latency” attribute computed as half the residual is an estimate, not a measurement. Reporting the residual itself, without halving it, is more honest, because the residual is directly measured and its interpretation is left to the reader.

Queue time is a duration, not a difference

One place where the temptation to subtract across hosts is strong and avoidable is queue latency. The time a message spends waiting can be computed two ways: as the difference between the consumer’s receive timestamp and the producer’s send timestamp, which crosses hosts and inherits the offset, or as a duration measured by the broker, which does not.

Where the broker records both enqueue and dequeue times, its own clock is the only one involved and the difference is sound. Where it does not, the cross-host difference is the only available estimate, and its error is the offset between producer and consumer. The distinction should be visible in the data model: a field named for a measured duration and a field named for a timestamp difference are not interchangeable, and collapsing them into one attribute makes the error bound unrecoverable.

The same reasoning applies to long-lived spans. A span that covers a queue wait and the processing that follows has a duration that is mostly waiting, and its start timestamp is the producer’s, so a reader comparing it to a consumer-side event is comparing across hosts. Separating the wait from the work, as two spans with a link or a parent-child pair, keeps each duration on one clock.

The general discipline is unglamorous: measure intervals on one clock, compare instants across clocks only with an explicit error bound, and record which is which in the field names. Sampling decisions, which are made in real time and often on a single host, are unaffected by all of this, but the traces they select inherit whatever timing error the collection pipeline carries; tail-based sampling in particular compares spans from many hosts inside one buffering window, which is where an unnoticed offset turns into a systematically wrong decision rule.