Measuring and Bounding Interrupt Latency

Interrupt latency is not one number. Between a device asserting a line and the thread that cares about the event being scheduled, there are at least five distinct delays, each caused by a different mechanism and each requiring a different fix. A measurement that reports a single figure without saying which of the five it covers is not a measurement of anything actionable.

The five are: the time until the processor takes the interrupt, which is set by whether interrupts are masked; the time until the handler runs, which is set by the entry path and by any higher-priority handler already executing; the time until the handler finishes, which is set by what the driver does in context; the time until the deferred work runs, which is set by the deferral mechanism chosen; and the time until a waiting thread is actually scheduled, which is set by the scheduler and by the priority of everything else runnable.

What masks the interrupt

The first delay is bounded by the longest region in which interrupts are disabled. On a uniprocessor this is the whole story, and the kernel’s discipline is to keep those regions short: a spinlock held with interrupts off should cover a few instructions, not a function call.

On a multiprocessor the situation changes. Disabling interrupts on one CPU does not stop another CPU from taking the interrupt, so the masking that matters is the masking on the CPU the interrupt is affinitised to. A driver that takes a spinlock with interrupts off on CPU 0 while the device is delivering to CPU 3 has not delayed the interrupt at all. The same code running on CPU 3 has.

This is why affinity and critical-section length interact, and why a system tuned for latency often pins interrupt sources to CPUs that run no latency-sensitive work. The pinning is not a guarantee, because the affinity can be overridden and the load balancer may move the source, but it converts an unpredictable delay into a predictable one.

The kernel measures the worst case directly. The latency tracer and the preemption-off critical section timing both record the longest interval with interrupts disabled, and the lockup detectors use the same infrastructure to warn when a CPU has been unable to take an interrupt for a suspicious duration. The soft lockup detector reports a CPU that has not scheduled; the hard lockup detector, driven by the non-maskable interrupt, reports a CPU that has not taken even that.

What runs before the handler

A higher-priority interrupt can preempt the entry path only if the gate allowed it. An interrupt gate clears the interrupt flag, so a second interrupt waits until the first handler executes sti. Most handlers do so early, and the deferred mechanisms are built to tolerate being interrupted.

The exception is the non-maskable interrupt, which cannot be masked by the flag at all. It is delivered regardless of the current state, which is what makes it usable for the hard lockup detector and unusable for anything that expects to run in a defined context.

The entry path itself has a cost that scales with what the handler does before it reaches the driver. Saving the full register set, switching stacks, and running the generic descriptor lookup are fixed costs, and their composition is set by the transition described in How a Hardware Interrupt Reaches a Handler on x86-64. On a machine with a large number of pending interrupts, the ordering between them matters more than the fixed cost: a low-priority source that is serviced first delays a high-priority one by the whole of its handler.

There is no software priority order among device interrupts. The local APIC does have one, but it is derived from the vector number itself: a vector’s priority is its top four bits, so vectors 0x30 to 0x3F outrank 0x40 to 0x4F, and an interrupt is held pending while the processor’s current task priority is higher. Its task priority register can raise that threshold and mask every vector below it, and the kernel uses it sparingly because a vector number is an allocation site, not a statement of importance: nothing about 0x31 makes the local APIC timer more urgent than 0x41 for a network queue. The ordering a kernel actually relies on is therefore the arrival order at the processor, with the hardware’s vector-number scheme as the only tie-breaker.

What the handler does

The third delay is entirely under the driver’s control, and it is the one that most often dominates. A handler that polls a status register, walks a descriptor ring, and copies data has a cost proportional to the ring size. A handler that loops until the device reports no more work has a cost proportional to the arrival rate, which is the runaway case that the budget in NAPI and Interrupt Mitigation exists to bound.

The measurement technique is to timestamp the entry and the exit and accumulate the difference. The tracepoints on the interrupt entry and exit paths provide exactly those two timestamps, and the histogram of differences is more informative than any summary statistic. A handler with a mean of two microseconds and a maximum of two milliseconds has a problem that the mean conceals: the maximum is the latency every other device on that CPU experienced.

The reason for a long tail in handler duration is usually a loop that is bounded by the device rather than by the code. A handler that processes “all completed descriptors” processes however many the device has finished, which under load is unbounded. A handler that processes at most N and leaves the rest for the poll is bounded. The distinction is a single loop condition, and it is the difference between a latency contributor and a latency source.

What the deferral costs

The fourth delay depends on the mechanism, and the four options described in Top Halves and Bottom Halves differ in what they wait for. A softirq runs at the tail of the interrupt that raised it, so its delay is the remainder of the handler plus the loop overhead: microseconds. A threaded handler waits for the kernel thread to be scheduled, which on an idle system is also microseconds and under load is not bounded. A workqueue item waits for a worker thread, and the workqueue’s concurrency limit can put it behind other items.

The scheduler’s involvement is what makes the fourth and fifth delays hard to separate in practice. A threaded handler that is ready to run but whose CPU is executing a lower-priority thread will not run until the scheduler switches, and the switch happens at a tick, at a wakeup that triggers preemption, or when the running thread blocks. The wakeup path does attempt to preempt immediately when the woken thread has higher priority, so the delay is usually one scheduling decision rather than one tick.

Priority inversion is the case where that reasoning fails. A high-priority thread waiting on a lock held by a low-priority thread is delayed by however long the low-priority thread runs, and if a medium-priority thread is runnable the low-priority one may not run at all. The kernel’s priority inheritance in rtmutexes addresses the mutex case; it does not address the case where the low-priority holder is in a softirq or a workqueue, because those are not threads with a priority that can be raised.

Observing the distribution

cyclictest is the standard tool for the fifth delay. It creates a thread that sleeps until an absolute deadline, records the difference between the deadline and the actual wakeup, and reports the distribution. The reported maximum is the number that matters for a real-time claim; the mean is almost meaningless, because a system that meets its deadline 99.9 percent of the time and misses it occasionally is not a real-time system.

The idiom it uses is a sleep against an absolute deadline rather than a relative delay, because a relative delay accumulates every error the previous iteration made. cyclictest implements the relative form as well, selectable at runtime, and it exists precisely so that the difference between the two can be demonstrated; the absolute form is what the default clock-nanosleep mode uses.

#define PERIOD_NS 1000000L                    /* one millisecond */

struct timespec next, now;
long long worst = 0;

clock_gettime(CLOCK_MONOTONIC, &next);
next.tv_sec++;                            /* the first deadline, one period out */

for (;;) {
    clock_nanosleep(CLOCK_MONOTONIC, TIMER_ABSTIME, &next, NULL);
    clock_gettime(CLOCK_MONOTONIC, &now);

    /* Overshoot in nanoseconds: how far the wakeup landed past the deadline. */
    long long overshoot = (long long)(now.tv_sec - next.tv_sec) * 1000000000LL
                        + (now.tv_nsec - next.tv_nsec);
    if (overshoot > worst)
        worst = overshoot;                /* the maximum, not the mean, is the number that matters */

    next.tv_nsec += PERIOD_NS;            /* advance the deadline, do not re-arm a delay */
    if (next.tv_nsec >= 1000000000L) {    /* carry nanoseconds into seconds */
        next.tv_nsec -= 1000000000L;
        next.tv_sec++;
    }
}

The measurement is only meaningful if the workload is representative. A test on an idle machine measures the timer and the scheduler and nothing else. A test with a competing CPU-bound load measures the scheduler’s preemption behaviour. A test with competing interrupt load measures the entry path. The three maxima are different, and a real-time design has to bound all of them.

The tracing infrastructure gives the other four delays. Function graph tracing with the interrupt entry and exit functions at the root gives the handler duration. The preemption and interrupt-disabled tracers give the masking intervals. The wakeup tracer gives the delay between a wakeup and the woken thread running. Each of these produces a histogram, and the histograms are what a latency budget is built from.

The cost of the tracing itself is a confounder. Enabling function tracing adds work to every traced function, which inflates the measured handler duration and can change the scheduling decisions that produce the delays being measured. The standard practice is to use the tracers to identify the mechanism and to use cyclictest or a dedicated hardware timestamp to quantify it.

Hardware assistance

The most direct measurement avoids the kernel entirely. A GPIO is toggled at the start of the handler and again at the end, and an oscilloscope measures the width. The same technique measures the delay from an external event to the handler: drive the GPIO from the event source and observe the delay to the toggle.

The advantage is that nothing in the measurement path can be scheduled, traced, or preempted. The disadvantage is that it requires hardware and that the toggle itself costs time, which is why the technique is used to calibrate the software measurements rather than as a routine check.

The processor’s own timestamp counter is the software equivalent, but reading it is not free and the read itself is affected by the same out-of-order execution that makes precise timing difficult. The kernel serialises the read where precision matters, at the cost of the serialisation.

What can be bounded, and what cannot

The delays that can be bounded by construction are the masking intervals, because they are code properties. The kernel’s real-time configuration reduces them in two directions. It converts most spinlocks to locks that can sleep, so a critical section no longer forces interrupts to stay masked while it runs. And it routes every interrupt handler through a kernel thread by default, so that the handler’s work happens in a schedulable context and the hard-irq stub that remains is only the mask-and-wake part. Together these remove the longest non-preemptible regions from the kernel, which is what a priority ceiling and a bounded number of such regions are measured against.

The delays that cannot be bounded by construction are those caused by other work. A handler’s duration depends on the device, and the scheduler’s decision depends on the load. A system can be designed so that these are usually small, and it can be instrumented so that they are observed when they are not, but it cannot be made to guarantee a bound without controlling the workload.

This is why real-time claims are always claims about a specific workload on a specific machine. The entry path, the deferral choice, the timer structure, and the polling budget each contribute a term to the total, and each term has a floor set by hardware and a tail set by contention.