Top Halves and Bottom Halves

An interrupt handler runs on a stack it did not allocate, in a thread it did not create, at a moment it did not choose. The interrupted context could be anything: a user process in the middle of a floating-point computation, another interrupt handler, the idle loop, or the scheduler holding a runqueue lock. The handler cannot block, cannot assume it owns any resource, and cannot rely on being scheduled again. These are not stylistic preferences; they follow from the fact that there is no task to block.

The work a device needs done is rarely compatible with those constraints. A network receive interrupt wants to allocate buffers, parse headers, and hand packets upward; a storage completion wants to wake a process. Some of that can be done in place, and some of it cannot. The division between the two is what “top half” and “bottom half” name, and the mechanisms on the deferred side differ mainly in what context they run in and what they are allowed to assume.

What interrupt context actually forbids

The kernel tracks interrupt context with a per-task preemption count. irq_enter increments it, irq_exit decrements it, and while it is nonzero the current task is marked as being in an interrupt. The scheduler checks this count before it will switch away, which is why a handler that calls a function capable of sleeping does not simply fail: it either triggers a warning from the sleeping-in-atomic checker or, if the count has been manipulated, corrupts state in a way that appears much later.

Sleeping means giving up the CPU and waiting to be rescheduled. That requires a task structure to save the context into, and an interrupt has none of its own. A handler can borrow the interrupted task’s context, but that task never agreed to be suspended inside a device driver, and its state may be arbitrary. So the rule is absolute: no sleeping in hard interrupt context.

The list of operations that can sleep is longer than it first appears. A mutex can sleep. A semaphore can sleep. kmalloc with GFP_KERNEL can reclaim and therefore sleep. copy_to_user can fault in a page and sleep. msleep, schedule, and anything that calls them sleep. Even a printk can block on a console lock, which is why the kernel has separate paths for logging from atomic context.

Non-sleeping alternatives exist for most of these: GFP_ATOMIC allocation, spinlocks instead of mutexes, and pre-filled buffers instead of user copies. What does not exist is a way to make an arbitrary long computation acceptable. Even a handler that never sleeps delays every other interrupt on the CPU and, if it runs long enough, delays the scheduler. A handler that takes a millisecond is a millisecond of added latency for everything else on that core.

Splitting at the acknowledge

The natural split point is the moment the device has been told its interrupt was seen. Until the acknowledge happens, the device may not be able to signal again, and a level-triggered line stays asserted. After it, the device is free and the remaining work is bookkeeping that does not need to happen immediately.

For a level-triggered line the kernel’s flow handler masks the line on entry and unmasks it after the action list completes, with one exception: if a threaded handler was woken on an IRQF_ONESHOT line, the unmask waits until that thread returns, so the line is quiet for the duration of the handler and of the thread. Otherwise the line is quiet only for the duration of the handler, regardless of what the driver does. For an edge-triggered line or an MSI there is nothing to mask, and the device will simply signal again if it has more work. The difference matters when the deferred work is slow: a masked level-triggered line loses no events, because the assertion persists, while an edge-triggered device may drop a second event that arrives while the first is still being processed. Hardware that can lose edges needs a status register that reports work still pending, which is why most such devices are polled after the handler rather than trusted to re-signal.

A driver that acknowledges early and defers everything else has a clean contract: the top half touches only registers that are safe to touch without locking, and the bottom half owns the data structures. A driver that acknowledges late has to be sure that the deferred work cannot depend on the device being unacknowledged.

The four deferral mechanisms

Softirqs are a fixed set of numbered contexts, declared at compile time, that run in interrupt context but with interrupts enabled on the local CPU. They are raised with raise_softirq, and the caller’s context decides where the work lands: raising one from interrupt context leaves the bit set for the tail of irq_exit to pick up, while raising one from process context wakes the per-CPU ksoftirqd thread, which runs the loop as an ordinary schedulable task. Because the set is fixed and small, the code can be tuned per instance, and because the handlers run with the interrupt flag off the preemption count but not in a task, they cannot sleep either. Networking receive processing is the largest consumer, and its budgeted version is described in NAPI and Interrupt Mitigation.

Tasklets were built on softirqs and provided a per-instance serialization guarantee: a given tasklet never runs concurrently with itself, even on different CPUs. The guarantee is enforced by a per-tasklet scheduled flag that the scheduling path sets with a test-and-set before enqueueing, so two CPUs cannot both place the same tasklet on their pending list. They have been deprecated in favour of threaded handlers and workqueues because the flag is only that: the work is still a softirq handler, with the same no-sleep constraint and the same latency, and the serialization buys nothing a threaded handler cannot provide more simply. Code that still uses them works; new code should not.

Workqueues run in process context, in kernel threads created by the workqueue subsystem. Work items may sleep, take mutexes, allocate with GFP_KERNEL, and block. The cost is a context switch and the possibility of delay: a workqueue thread is scheduled like any other thread, so under load the work may wait. Workqueues also offer concurrency management, ordering guarantees between items placed on the same worker, and the ability to flush a queue before teardown.

Threaded interrupt handlers are the fourth option and the one that most directly matches the split. request_threaded_irq takes two functions: a primary handler that runs in hard interrupt context and is expected to do the acknowledge, and a thread function that runs as a real kernel thread with the interrupt’s priority. The primary returns IRQ_WAKE_THREAD to hand off, or IRQ_HANDLED to finish in place. With IRQF_ONESHOT the line stays masked until the thread completes, which is what a level-triggered device needs when the thread is the only thing that can clear the condition.

static irqreturn_t demo_primary(int irq, void *dev)
{
    struct demo_dev *d = dev;
    u32 status = readl(d->base + REG_STATUS);

    if (!(status & STATUS_IRQ))
        return IRQ_NONE;              /* not our device: shared line */

    writel(status, d->base + REG_ACK); /* acknowledge in hardware */
    return IRQ_WAKE_THREAD;            /* hand the rest to the thread */
}

static irqreturn_t demo_thread(int irq, void *dev)
{
    struct demo_dev *d = dev;

    mutex_lock(&d->lock);              /* legal: process context */
    demo_process(d);
    mutex_unlock(&d->lock);
    return IRQ_HANDLED;
}

Choosing between them

The axes are latency, context, and serialization.

Latency to first work: a softirq runs before returning to the interrupted task, so its latency is bounded by the handler itself. A workqueue item waits for a thread to be scheduled, which on an idle system is short and on a loaded system is not bounded. A threaded handler sits between the two, since the kernel wakes the thread immediately but the thread still has to be scheduled.

Context: softirqs and tasklets cannot sleep. Workqueues and threaded handlers can. The choice is often forced by whether the deferred work needs a mutex or an allocation that may reclaim.

Serialization: softirqs of the same number can run concurrently on different CPUs, so per-CPU data is required unless the code takes a lock. Tasklets serialize per instance. Workqueue items serialize only if placed on an ordered queue or if the queue’s max-active is one. Threaded handlers for the same line serialize naturally, because the line is masked or the thread is not re-woken until it returns.

The common mistake is to treat the choice as a performance decision when it is a correctness decision. A driver that needs a mutex has exactly one option among the four. A driver that needs sub-microsecond latency has one option. The interesting cases are the ones where both constraints are absent, and there the tie-breaker is usually how much of the work is per-CPU: softirqs give natural per-CPU locality, workqueues do not unless bound.

The ksoftirqd escape hatch

Softirqs cannot run unbounded, because they run before returning to the interrupted task and would starve it. The kernel enforces two limits. The first is a per-invocation time budget in the softirq loop; when it is exceeded, the remaining work is handed to ksoftirqd, a per-CPU kernel thread. The second is a restart limit that prevents a softirq which re-raises itself from looping forever.

The consequence is that softirq processing can migrate from interrupt context to process context mid-stream. Code that runs in a softirq must therefore not assume it is in interrupt context, only that it cannot sleep. The distinction is invisible to most drivers and critical to the ones that inspect the preemption count.

ksoftirqd also explains a common observation: under sustained interrupt load, a per-CPU kernel thread appears near the top of a process listing. That thread is not the interrupt handler; it is the deferred remainder, and its presence indicates that softirq processing is being budgeted rather than that interrupts are being lost.

Both halves misbehaving

Deferring too much is a failure. A driver that acknowledges in the top half and does everything in a workqueue converts a bounded interrupt into unbounded queue latency, and under load the queue grows faster than it drains. The visible symptom is a device that reports completion long after the operation, or a socket receive queue that overflows.

Deferring too little is also a failure. A handler that processes a full packet in hard interrupt context holds the CPU against every other interrupt for the duration. On a machine with several high-rate devices this produces a latency distribution with a long tail that no amount of scheduling can fix.

Sharing state between halves without a lock is the third. The top half and the bottom half for the same device can run concurrently on different CPUs if the bottom half is a softirq, so any structure they both touch needs a spinlock, and the top half needs the irqsave variant since it may already have interrupts disabled.

Freeing the device while deferred work is queued is the fourth. Teardown must cancel or flush pending work before releasing the device’s memory. On the interrupt side, free_irq synchronises against the hard handler by waiting for any in-flight interrupt on other CPUs and by releasing the line; on the deferred side, the workqueue layer has an explicit flush that waits for queued items to finish, and a threaded handler is torn down by the same function that created it. The two are separate guarantees, and a driver that frees first and flushes later has a window in which the deferred work touches freed memory.

The mechanisms here are the middle of the cluster’s chain. Above them, the entry path and its constraints are set by the hardware transition in How a Hardware Interrupt Reaches a Handler on x86-64. Below them, the softirq machinery itself, its per-CPU queues, and the budgeting rules are examined in Softirqs and the Per-CPU Deferred Queue.