Clock Domains and the Ordering of Events Across Hosts

A request arrives at a gateway at 12:00:00.100 and the response leaves a backend at 12:00:00.350. The difference, 250 milliseconds, looks like a measurement. It is not: it is the difference between two readings of two different clocks, and its error is the sum of both clocks’ offsets from true time plus the error in each reading. On hosts synchronised by NTP over the public internet, offsets of a few milliseconds are ordinary and offsets of tens of milliseconds are possible; on virtual machines whose clocks are disciplined differently from the host, worse. Any conclusion drawn from a cross-host timestamp subtraction inherits that error, and a latency budget of 5 milliseconds cannot be investigated with an instrument that has 20 milliseconds of offset. The same bound applies to every correlation between a change and a symptom, which is why onset estimation has to work at a resolution coarser than the clock error it is built on.

Two clocks per machine, and only one of them measures duration

Every Unix-like system exposes at least two notions of time through clock_gettime. CLOCK_REALTIME reports wall-clock time as a count of seconds and nanoseconds since the Unix epoch, and it is what timestamps in logs and trace records carry. CLOCK_MONOTONIC reports time since an unspecified starting point, typically boot, and it never runs backwards. On Linux, CLOCK_MONOTONIC_RAW additionally excludes the frequency adjustments NTP applies, and CLOCK_BOOTTIME includes time spent suspended.

The distinction that matters for debugging is that realtime can jump and monotonic cannot. A duration measured as the difference of two CLOCK_MONOTONIC readings is correct even if the machine’s wall clock is wrong by an hour, because the same wrong clock is subtracted from itself. A duration measured as the difference of two CLOCK_REALTIME readings is correct only if the clock did not change between them, and NTP changes it in two ways: slewing, which adjusts the rate by at most about 500 parts per million so the clock converges gradually, and stepping, which sets the clock to a new value in one operation.

Slewing distorts durations proportionally. A process that takes one second of true time while the clock is being slewed by 500 ppm is recorded as taking between 0.9995 and 1.0005 seconds, an error of half a millisecond per second. That is irrelevant for a request and significant for a job that runs for an hour, where the accumulated error reaches roughly 1.8 seconds. Stepping is worse: a backward step of 30 milliseconds in the middle of a measurement produces a negative duration, and a forward step produces a duration inflated by the step size. Both appear in trace data as impossible spans, and neither indicates a bug in the application.

Go’s time.Now returns a Time that carries both a wall-clock reading and a monotonic reading, and the Sub method uses the monotonic component when both operands have one. That is why time.Since(start) is safe and end.UnixNano() - start.UnixNano() is not. The language-level fix exists because the problem is common enough to have caused silent misreporting in programs that looked correct.

func measure() (time.Duration, int64) {
	start := time.Now()                    // carries a monotonic reading
	work()                                 // whatever the operation is
	elapsed := time.Since(start)           // monotonic: immune to wall-clock steps

	// Converting first discards the monotonic reading, so a step during
	// work() shows up here as an inflated or negative duration.
	stamped := time.Now().UnixNano() - start.UnixNano()
	return elapsed, stamped
}

The same trap exists in languages without a combined representation. In C, taking CLOCK_REALTIME for a duration measurement is a choice, and the correct choice is CLOCK_MONOTONIC; the fact that the wrong choice compiles and usually produces plausible numbers is what makes it survive review.

Offset, skew, and the difference between them

Offset is the instantaneous difference between a host’s clock and a reference. Skew is the rate at which that offset changes, usually expressed in parts per million; a typical crystal oscillator is specified within roughly 20 to 50 ppm, meaning a day of free-running time accumulates between about 1.7 and 4.3 seconds of drift. NTP estimates both: it measures the round trip to a server, assumes the path is symmetric, and computes offset as half the difference between the two one-way estimates, with an error bound derived from the round-trip time and the server’s own stratum.

The error bound is the part most often dropped. NTP’s root distance combines the round-trip delay, the dispersion accumulated along the synchronisation chain, and the server’s claimed precision. A host synchronised to a stratum-2 server over a congested path can legitimately have an offset error bound of several milliseconds even while its reported offset is near zero. Two such hosts can be offset from each other by twice that bound, in opposite directions, and their difference of timestamps carries the sum.

There is a further wrinkle in virtualised environments. A guest’s clock is usually disciplined by the hypervisor through a paravirtualised clock source, and that source may or may not track the host’s adjustments. Live migration can move a guest to a host whose clock differs, producing a step that the guest sees as an unexplained discontinuity. Time in a container follows the host kernel, so containers on the same host share a clock and containers on different hosts do not, a distinction that matters when the two are assumed to be comparable.

Ordering without a shared clock

The alternative to trusting timestamps is to record causality. Lamport’s insight is that a distributed system can be described by a partial order: if event $a$ happens before event $b$ in the same process, or if $a$ is a message send and $b$ the corresponding receive, then $a$ precedes $b$. Events with no such path are concurrent, and no clock is needed to say so.

Lamport clocks implement this with a counter per process: increment on each local event, and on receive set the counter to one more than the maximum of the local value and the received value. The resulting numbers are consistent with causality — $a \to b$ implies $L(a) < L(b)$ — but the converse fails, since a larger counter does not imply a causal path. That one-way property is enough for many purposes and insufficient for detecting concurrency.

Vector clocks fix the converse by carrying one counter per process. Event $a$ precedes $b$ exactly when every component of $a$‘s vector is less than or equal to $b$‘s and at least one is strictly less. The cost is $O(n)$ state per event in the number of processes, which is why wide systems prefer other structures. Hybrid logical clocks, used by several databases, combine a physical reading with a logical counter so that timestamps remain close to wall time while still preserving the causality property, and their size stays constant.

Tracing systems sidestep the problem rather than solve it. A span carries a start and end recorded by its own host’s clock, and the parent-child relationship is carried explicitly in the record rather than inferred from timestamps. The trace’s structure is therefore a causal order, and the durations are locally measured. What the trace does not give is a trustworthy global timeline: drawing spans on a shared axis requires estimating each host’s offset, and any error in that estimate appears as a shift of an entire service’s spans relative to its neighbours.

Reading a trace that contains an impossible span

A child span that starts before its parent, or ends after it, is the standard symptom of offset. The naive reading is that the instrumented code is broken, and the correct first question is whether the two spans were recorded on the same host. Same-host impossible spans indicate a real problem: a duration computed from a wall clock that stepped, or a span started in one goroutine and ended in another after a context switch that crossed a clock adjustment.

Different-host impossible spans usually indicate offset, and the magnitude of the impossibility bounds the offset. If a child appears to start 40 milliseconds before its parent, the two clocks differ by at least 40 milliseconds, assuming both durations are accurate. That bound is useful: it tells the investigator whether the anomaly is large enough to explain the discrepancy being studied, or whether the discrepancy must have another cause.

There is a second diagnostic use. When a cross-host latency looks implausibly large but each host’s internal measurements are consistent, the gap between client-observed and server-observed time is the network and queueing path, and its size can be estimated without trusting either clock, by comparing durations rather than timestamps. The client’s total duration minus the server’s handling duration leaves the time spent in transit, retries, connection setup, and queueing before the request reached the handler. Both durations are locally measured and therefore offset-free.

Instrumenting so that the question stays answerable

The design constraint is that timestamps recorded once, at one host, are cheap; timestamps that can be compared across hosts require either synchronisation infrastructure with a known error bound or an explicit offset-estimation step in the analysis pipeline. The trade is between trusting the clock and paying to verify it.

Three practices follow. Record durations from monotonic sources and keep the raw timestamps only for ordering within a host. Record the clock source and its estimated error alongside the timestamp when the error bound is known, since a timestamp without a bound invites the reader to treat it as exact. And prefer causal evidence over temporal evidence when both are available: a log line written by the callee before it returns is a stronger statement about ordering than any comparison of two clocks, because it does not depend on either one.

The limits are real. Causality gives order but not magnitude, so it cannot answer “how long did this take” across hosts. Offset estimation gives magnitude with a confidence interval that is often wider than the effect under investigation. Choosing between them is a question about what the investigation needs, and the answer changes as the investigation narrows, which is why discriminating tests should specify which of the two kinds of evidence would be decisive before the probe is run.