Clock Skew Injection and the Failures That Depend on Wall Time

Time is the fault domain that most systems implicitly trust. Every other injected fault has a visible signature — a missing process, a slow response, a closed socket — but a clock that is wrong produces a system that looks entirely healthy and makes decisions based on an incorrect premise. Sequences that were ordered are no longer ordered, credentials that were valid are not, and leases that were held are considered abandoned.

Offset, skew, and drift are three different faults

The instantaneous difference between a node’s clock and a reference is the offset. The rate at which that difference is changing is the skew, normally expressed in parts per million: a skew of 1000 ppm means the clock gains or loses about 86 seconds per day. Drift is the slow, thermally and electrically influenced variation of the hardware oscillator around its nominal frequency, and it is the reason offset correction alone does not converge. Synchronisation protocols estimate all three: a stepping correction jumps the offset to zero, while a slewing correction adjusts the effective frequency so the offset is driven out gradually.

An injection has to pick one of these, because the three have different consequences. An offset fault is a single discontinuity, and the interesting question is what was in flight across it. A skew fault changes the rate of every interval measurement, and the interesting question is which arithmetic is sensitive to the length of a second. Drift with no correction is the slowest and most dangerous, since nothing about the system appears broken at any particular moment, and the fault only becomes visible when enough accumulated difference crosses a threshold such as a validity window.

The magnitude that matters depends on the tolerance of the consumers. Kerberos specifies an allowable clock skew with a default of five minutes between the parties to an authentication exchange, which sets a natural ceiling for a clock experiment on an authentication path. Certificate validity windows are measured in days or months, so only a large step or a very long-running drift reaches them. Neither number is a safe default for a given system; both are examples of tolerances that exist somewhere in the design and are rarely written down. The magnitude of a clock experiment is therefore derived rather than chosen: the tolerance belongs in the bound of the steady-state hypothesis that the experiment is testing, and a fault well below it tests nothing while a fault far above it tests only the first validation check on the path.

Stepping the kernel clock, or interposing on one process

The blunt instrument sets the system clock. That requires the privilege to do so, affects every process on the host, and is awkward to undo cleanly, because the moment the clock is restored, any timer that already fired is in the past. Stepping forward past a scheduled wakeup causes the kernel to service that timer immediately, so a batch of deferred work executes in a burst; stepping backward delays timers, and some clock APIs can report a deadline that has already expired. Sleeping until an absolute wall-clock time is undefined across a step in a way that is not documented at the call site.

A time namespace offers a contained version of the step for the clocks it covers. A namespace carries per-namespace offsets for the monotonic and boot-time clocks, visible for a process through its own reporting path, and the offset does not affect the real-time clock, which stays shared with the host. This is a genuinely useful separation: it permits an experiment about timer and interval behaviour without disturbing the system’s notion of the date, which is what most credential checks use.

The third approach interposes between the program and the clock at the C library boundary, replacing the entry points for reading the time so they return a shifted value. Its limitation is structural rather than incidental. Programs that read the clock through the kernel’s fast path, or that issue the system call directly rather than through the C library, bypass the interposition entirely, as do statically linked binaries and runtimes that implement their own time handling. An experiment using this method measures the subset of the program that goes through the interposed layer, and the subset that does not keeps the correct time, which produces a system whose internal clocks disagree with each other. That disagreement is a fault in its own right, and one that is worth testing deliberately, but it is not the same fault as a wrong host clock.

What breaks when wall time moves

The failures cluster into four families. Validation windows are the first: a certificate is rejected outside its validity period, and a token whose expiry is expressed as an absolute instant is rejected once the clock passes it. Both checks are synchronous and immediate, so a forward step of sufficient size turns authenticated traffic into rejected traffic within one request.

Duration arithmetic is the second, and it is worse because it is silent. A difference between two wall-clock readings is not guaranteed to be positive: a deadline computed as an expiry minus the current time can go negative, a rate limiter that refills a bucket from the elapsed wall time can be handed a negative interval, and a cache whose entries carry a timestamp can serve entries for longer than intended if the clock moves backward. Code that measures how long an operation took using a wall clock can produce negative latencies, which then propagate into trace timelines and percentile calculations.

Ordering is the third. Timestamps assigned by different nodes are not comparable when their clocks differ, so log and event streams interleave incorrectly and a causal ordering reconstructed from timestamps can place a consequence before its cause. Distributed systems that need a real ordering use logical constructs — sequence numbers, versions, terms — precisely because physical time cannot supply one; the physical timestamp is metadata, not evidence.

Leases and elections are the fourth, and they are the most consequential because they are the mechanism by which the system avoids having two leaders at once. A leader that grants itself a lease based on its own clock, and a follower that decides the lease has expired based on a differently wrong clock, can both believe they are entitled to act. Well-designed lease protocols bound how much the two clocks can differ and refuse to operate outside that bound, which converts clock error into unavailability instead of into split brain. That bound is a testable assumption and a natural input to the experiment’s magnitude: a skew just below it should be tolerated, and a skew just above it should produce the protocol’s defined refusal.

#include <stdio.h>
#include <time.h>

int main(void) {
    struct timespec real, mono, a, b;
    clock_gettime(CLOCK_REALTIME, &real);
    clock_gettime(CLOCK_MONOTONIC, &mono);
    /* the difference between these two is an arbitrary large number that
       changes with every offset adjustment; only each clock's own delta matters */
    printf("wall %lld, monotonic %lld\n",
           (long long)real.tv_sec, (long long)mono.tv_sec);

    clock_gettime(CLOCK_MONOTONIC, &a);
    /* work that must be measured with a clock that cannot go backwards */
    clock_gettime(CLOCK_MONOTONIC, &b);
    printf("elapsed %lld ns\n",
           (long long)(b.tv_sec - a.tv_sec) * 1000000000LL +
           (long long)(b.tv_nsec - a.tv_nsec));
    return 0;
}

Why the monotonic clocks exist, and where they still lie

The monotonic clock advances at a rate that does not depend on synchronisation adjustments and never goes backward, which is exactly what interval measurement needs. It also does not advance while the machine is suspended, since it counts running time rather than elapsed existence; a separate boot-time clock continues across suspend and is the right choice for measuring wall-clock-equivalent durations across a sleep. Choosing between them is a statement about what is being measured, and choosing the real-time clock for a duration is almost always a bug that only a clock fault reveals.

Even monotonic clocks have edges. They are not comparable across processes on different machines, they have no defined epoch, and they reset on reboot, so a value cannot be persisted and compared later. Within a time namespace they carry an offset, so two processes on the same host can disagree about the same monotonic instant when they are in different namespaces, which is a legitimate configuration and a nasty surprise for anything that assumes host-wide agreement.

Reverting a clock fault deserves the same care as reverting a kill. Restoring the host clock is another step, and a large backward step can re-fire code paths that treat the past as an anomaly. The safest sequence follows the general rule for any injection: the fault should be applied to a bounded target set, described in the same units as the abort condition, and reverted through a path that was exercised before the fault began. The techniques for keeping the exercise’s own traffic and telemetry distinguishable from real incidents, described in marking injected faults in telemetry, matter more here than anywhere else, because a clock fault can corrupt the timestamps that would otherwise be used to reconstruct what happened.