Clock Sources and the Resolution Floor of a Timing Call

An interval is the difference between two readings of a clock, and every statement about how long something took inherits the error of both readings. The error has two parts: a resolution floor, meaning the smallest distinguishable step, and a read cost, meaning the work the clock itself performs inside the region being measured. Both are properties of the clock source, and the choice of source silently bounds what the rest of the benchmark can conclude.

The clock is inside the region it measures

Timing a region requires reading the clock before it and after it. Those reads are themselves instructions that execute, occupy issue slots, and touch memory or registers. On a modern out-of-order core a read of a hardware counter may retire in tens of cycles, while a call that enters the kernel to read a maintained time structure costs hundreds to a few thousand cycles depending on whether it is satisfied by a vDSO mapping or a real trap.

The relative error is therefore $2c/I$ where $c$ is the read cost and $I$ is the measured interval. At an interval of a million cycles and a read cost of fifty, the overhead is one part in ten thousand and can be ignored. At an interval of a thousand cycles the same two reads are a ten percent tax, and worse, they may prevent the very effect under study by displacing the measured instructions from the front end of the pipeline.

Amortisation is the standard answer: time a loop of $n$ iterations rather than one, and divide. This works only if the iterations are independent and identical. When they are not, the loop average is a weighted mixture, and the weighting is not the one a real workload would produce.

Monotonic, realtime, and what each guarantees

POSIX defines several clocks with different contracts. CLOCK_REALTIME represents wall-clock time and is subject to adjustment by the system administrator and by time synchronisation: it can jump forwards or backwards, so an interval computed from two readings can be negative. CLOCK_MONOTONIC never runs backwards and is not affected by adjustments to the wall clock, which makes it the correct source for measuring elapsed time. CLOCK_MONOTONIC_RAW on Linux exposes a counter that is not slewed by NTP correction, trading long-term drift relative to real time for immunity to the small frequency adjustments that slewing applies.

The distinction matters at the scale of a long benchmark run. A clock being slewed is deliberately run slightly fast or slow to bring the system time into agreement with the reference; the adjustment is temporary, but the residual frequency error that remains after a correction is not, and a residual of a few parts per million accumulates a few milliseconds per hour because one part per million is about 3.6 milliseconds per hour. During a correction the rate itself is changed, by hundreds of parts per million on a large offset, which moves intervals measured inside that window relative to intervals measured outside it.

CLOCK_PROCESS_CPUTIME_ID and CLOCK_THREAD_CPUTIME_ID measure CPU time consumed rather than elapsed time, excluding periods when the thread was descheduled. For single-threaded compute-bound work the two are close, and CPU time removes scheduler noise. For work that blocks, CPU time measures something different from latency and is the wrong instrument if latency is the question.

Reading the counter without a syscall

The kernel maintains time in memory that user space can read directly. On x86-64 Linux, clock_gettime for a coarse or monotonic clock is served by a vDSO routine mapped into the process, so the call does not enter the kernel: it reads a shared data structure and does the arithmetic in user mode. The cost is then on the order of tens of nanoseconds rather than the roughly few hundred to a thousand nanoseconds a genuine syscall costs on current hardware.

A coarser option exists: CLOCK_MONOTONIC_COARSE reads a value the kernel updates at the tick rate, so it is far cheaper but quantised to the tick, typically one millisecond or a small multiple of it depending on configuration. A benchmark that times sub-millisecond operations with the coarse clock will report a distribution of zeros and tick multiples, which is the signature of quantisation rather than of the code.

The processor’s time-stamp counter is cheaper still, a single instruction. Its use for measurement carries three problems. It counts at a constant reference frequency that is not necessarily the core’s current frequency, so it measures reference cycles rather than core cycles, which is actually what a frequency-invariant comparison wants but surprises people expecting core cycles. It is not synchronised across sockets on all systems. And it is not serialising: out-of-order execution can move the read earlier or later relative to the instructions of interest, so it must be bracketed by a serialising instruction such as lfence or by the rdtscp variant that also reports the processor identifier. The serialisation adds its own cost and constrains instruction scheduling inside the region.

The quantisation of the reading, and of the interval

A clock with resolution $\delta$ reports readings on a grid of width $\delta$. An interval computed from two grid readings is itself on that grid, and the rounding errors of the two readings combine. Under the usual assumption of independent uniform rounding errors, the error of the difference has standard deviation $\delta/\sqrt{6}$, which is about $0.41\delta$. The practical consequence is that intervals much shorter than $\delta$ cannot be distinguished at all, and intervals a few multiples of $\delta$ carry a relative error of tens of percent even with no other noise source.

Repeated timing of the same fixed work does not average this away in the way it averages other noise, because the rounding error is not independent of the work: a loop whose iterations take a constant number of cycles lands on the same grid positions each time. The measured value can be stable to the digit and still be wrong by up to one grid step, a bias that repetition cannot detect. Only changing the clock or lengthening the measured region reveals it.

#include <time.h>
#include <stdio.h>

/* Measure the resolution of a clock by observing the smallest nonzero delta. */
int main(void) {
    struct timespec a, b;
    long best = -1;
    for (int i = 0; i < 100000; i++) {
        clock_gettime(CLOCK_MONOTONIC, &a);
        do {
            clock_gettime(CLOCK_MONOTONIC, &b);
        } while (b.tv_sec == a.tv_sec && b.tv_nsec == a.tv_nsec);
        long d = (b.tv_sec - a.tv_sec) * 1000000000L + (b.tv_nsec - a.tv_nsec);
        if (best < 0 || d < best) best = d;
    }
    printf("smallest observed step: %ld ns\n", best);
    return 0;
}

The smallest observed step is an upper bound on resolution, not the resolution itself: the loop cannot read faster than the read cost allows, so on a fine-grained clock the observed step is dominated by the cost of the read. The figure is still useful, because it is the floor the harness actually operates at.

Resolution is not accuracy

A clock can have nanosecond resolution and be wrong by milliseconds. Resolution bounds how finely differences can be expressed; accuracy bounds how close the reading is to the quantity it claims to represent. For interval measurement only the rate matters, not the offset: a clock that is uniformly five milliseconds fast still measures intervals correctly. What breaks intervals is a rate that changes during the run, which is exactly what frequency scaling does to the time-stamp counter if the counter is derived from the core clock rather than from a fixed reference, and what NTP slewing does to the monotonic clock at the tens-of-ppm level.

Virtual machines add a further layer. A guest reading a paravirtualised clock is reading a value maintained by the hypervisor, and the cost of that read depends on whether the hypervisor handles it without a VM exit. When it does not, a timing call becomes a trap whose cost is in the microseconds and whose variance is large, which places a floor under every measurement taken inside the guest that no amount of repetition removes.

Choosing the instrument to match the question

The selection rule follows from the error model rather than from preference. For intervals of milliseconds and above, a monotonic clock read through the vDSO is cheap enough to ignore and fine enough to resolve the interesting variation. For intervals of microseconds, the read cost becomes a measurable fraction and the harness should amortise over a loop. For intervals of tens of nanoseconds, the clock read is comparable to the work and the correct approach is to measure a batch and subtract a measured empty-batch cost, or to count events instead of time.

Counting is often the better instrument. A hardware performance counter or a software counter incremented per operation gives an exact integer with no resolution floor at all, at the cost of a different question being answered: counts of instructions or cache misses are not time, and their relationship to time depends on the frequency and on the overlap the core achieves. Measurement Overhead and the Code the Compiler Deletes describes what happens when the timed region is optimised out entirely, and Warmup Transients and Detecting Steady State in Timed Loops describes how the choice of clock shapes the transient the harness has to discard.