Clock Skew and the Limits of Cross-Host Event Ordering

Merging event logs from several machines produces a single sorted timeline, and the sort is the most dangerous operation in an investigation. Timestamps recorded by different hosts are points drawn from different, drifting time bases, and a sort silently converts those points into a total order that may not match what happened. The error is not random noise that averages out across many events; it is a per-host offset that persists and that shifts the entire host’s contribution to the timeline relative to its neighbours.

This matters because the conclusions most investigations rest on are ordinal: process A ran before process B, the login preceded the file write, the outbound connection happened after the configuration change. Getting the offsets wrong can invert any of those.

Two clocks, only one of which is easy to record

A Unix host has several independent time sources exposed through clock_gettime. CLOCK_REALTIME is the calendar clock, the one adjusted by NTP and the one most log formats store. CLOCK_MONOTONIC counts time since an unspecified starting point, is never adjusted backwards, and is unaffected by NTP steps. CLOCK_BOOTTIME is like monotonic but includes time spent suspended.

Realtime is adjustable in two distinct ways, and the distinction is what breaks timelines. A step sets the clock abruptly, producing a discontinuity: timestamps jump forward or back by the correction amount. A slew changes the rate at which the clock advances until the error is absorbed, producing no discontinuity but a stretch of time during which the clock runs measurably fast or slow. Both are normal operation for any daemon implementing RFC 5905. An investigator reading two log lines on either side of a step sees an interval that never existed, and one reading lines during a slew sees an interval that is slightly wrong in a way no record flags.

The saving grace is that on most Linux systems the kernel logs the adjustment itself. The kernel ring buffer receives messages when the clock is stepped or when it enters and leaves a leap-second handling state, and the NTP daemon’s own log records each chosen offset and each adjustment it applies. Those records are the only way to recover the mapping between realtime and the underlying progression of time.

How much error is plausible

NTP-synchronized hosts on a well-behaved local network converge to offsets in the low single-digit milliseconds or better; servers a few network hops from an authoritative source commonly hold within a few milliseconds, and hosts syncing across a wide-area path within tens of milliseconds. Virtual machines complicate this: a guest clock may be disciplined by the host through a paravirtual mechanism in addition to NTP, and a guest that was suspended and resumed can come back with a large instantaneous error. Systems with no time daemon at all drift at rates set by the quartz oscillator, commonly a few seconds per day and, on cheap hardware or under heavy interrupt load, substantially more.

Those figures translate directly into an ordering uncertainty. If two hosts are each within 10 ms of true time, an event on one cannot be ordered against an event on the other with confidence unless the recorded times differ by more than the combined 20 ms. Anything closer is a tie. Building the timeline means computing, per host, an uncertainty interval, and comparing intervals rather than points.

Constructing an interval-aware timeline

The practical representation is a list of events each carrying a [earliest, latest] window rather than a single instant. Sorting by either bound produces a defensible ordering of the windows, and overlapping windows are reported as concurrent rather than ordered.

import sqlite3

def insert_event(conn, host, ts_epoch, uncertainty_s, source, detail):
    # Window reflects the host's worst plausible offset at that moment.
    conn.execute(
        "INSERT INTO events (host, lo, hi, source, detail) VALUES (?, ?, ?, ?, ?)",
        (host, ts_epoch - uncertainty_s, ts_epoch + uncertainty_s, source, detail),
    )

def order(conn):
    rows = conn.execute(
        "SELECT host, lo, hi, source, detail FROM events ORDER BY lo, hi"
    ).fetchall()
    out, horizon = [], None
    for host, lo, hi, source, detail in rows:
        if horizon is None or lo > horizon:
            out.append(("ordered", detail))
        else:
            out.append(("concurrent", detail))
        horizon = hi if horizon is None else max(horizon, hi)
    return out

The concurrent label is not a failure of the analysis. It is the honest statement that the sources cannot distinguish the two events, and it prevents the most common investigative error: narrating a causal chain that the evidence supports only if a sub-second ordering is assumed.

The sources that sidestep the problem

Some record types carry a clock that is independent of the host’s realtime. Network captures taken by a tap or span port are stamped by the capture device, so all packets in one capture share a single time base regardless of how many hosts they involve. If that capture device is synchronized, cross-host correlation through the network is exact to the capture resolution, and the host logs can be anchored to it. The same applies to a log forwarder that stamps arrival time at a centralized collector: the arrival stamp is a single-host clock, but it bounds the event from above and is immune to the sending host’s drift.

Sequence numbers provide ordering without any clock at all. TCP sequence numbers order bytes within a connection; a monotonic counter in an application log orders records within a stream. Where a monotonic counter exists, ordering is exact and timestamps become descriptive rather than structural, which is why authentication logs that carry both a local timestamp and a unique identifier are more useful than either alone, a point that recurs in lateral movement from authentication logs.

What survives, and what does not

Coarse conclusions survive badly-chosen clocks remarkably well. Minutes-scale separation on hosts with milliseconds of error is safe. Day-scale reasoning is safe even for hosts with no synchronization. The fragile cases are the ones that matter most: distinguishing an automated login from the action that followed it, or deciding whether a file was read before or after a packet left the host.

Leap seconds are a separate source of error that deserves naming, because they produce an off-by-one-second discontinuity rather than a drift. The standard handling in a kernel is to slew the clock across a window or to step it, and which of the two applies depends on configuration. Hosts in the same environment configured differently will disagree by exactly one second for the duration of the slewing window, which is large enough to invert an ordering that a naive sort treats as established.

The recoverable line of attack is to distrust the primary timeline’s fine structure and rebuild it from a source with a single clock. Capture-based ordering, monotonic counters, and parent-child inheritance of a single host’s timestamps all do this. For the parts of the timeline that must be reconstructed from killed processes and their parentage, the inheritance property described in process start records is more reliable than any wall-clock comparison, because an inherited timestamp is copied rather than re-read from a drifting source.

Where no single-clock source exists for a segment of the timeline, the honest presentation is a partial order rather than a sequence. A partial order states which pairs of events are ordered and which are not, and it is a stronger claim than a total order built on assumed offsets because it does not assert anything the evidence cannot support. Tools that render partial orders as parallel lanes, with concurrency visible, express the uncertainty directly rather than burying it in a millisecond value.

One further correction is worth stating because it is easy to get wrong: converting timestamps to a reporting timezone is a presentation step and must happen after correlation, never before. A timeline assembled in local time across hosts in different zones has both drift and an unknown offset folded into it, and untangling the two afterwards is usually impossible. The same applies to daylight-saving transitions, which produce an hour-long backward jump in local time and therefore an apparent ordering violation in any timeline sorted before conversion.