Reboot-Required and Live Kernel Patching

A patched machine and a running machine are different objects. The package transaction writes new bytes to disk and reports success; the processes that were started before the transaction continue to execute the bytes they mapped at startup. For most daemons this is harmless, because the next restart picks up the new code. For the kernel it is not, because the kernel cannot be restarted without restarting everything else. The reboot flag exists to record that gap, and it is a much weaker statement than most people reading it assume.

One flag, several distinct facts

The convention on Debian-derived systems is a file at /var/run/reboot-required, written by maintainer scripts when a package decides that its changes cannot take effect without a restart. On a system where /var/run is a tmpfs, the file disappears at boot, so its presence is scoped to the current boot by construction. A companion file, /var/run/reboot-required.pkgs, lists the packages that asked for the flag, which is the difference between “something wants a reboot” and “the kernel wants a reboot”.

The flag is advisory. Nothing enforces it, no service refuses to start because of it, and a package that forgets to write it produces a machine that is silently running stale code. It is also coarse: it collapses at least three distinct conditions into one bit.

The first condition is a new kernel image on disk. The running kernel is identified by the release string that uname -r reports, and modules live under /lib/modules/<release>. Installing a kernel package adds a new directory and leaves the old one in place, which is why the previous kernel remains bootable. The running kernel has not changed. Anything that loads a module after the upgrade resolves it against the running release’s directory, which still exists and still contains the old modules, so module loads keep succeeding even though the installed kernel is newer.

The second condition is a module whose file has been replaced for the running release. This is the case that breaks quietly: a module loaded at boot stays in memory, and a module loaded later comes from the new file. Two modules in the same kernel can therefore be from different builds. When a distribution ships a kernel update, it normally ships a new release string precisely to avoid this, but out-of-tree modules and firmware packages can create it.

The third condition is userspace. A library or an interpreter that has been replaced will not be seen by processes that already mapped the old file, and the flag is often set for exactly this reason. That condition has no live fix at all.

Why a running process keeps the old library

The package manager does not rewrite a file in place. It unlinks the old path and creates a new file, or writes a new file and renames it over the old name. Either way the old inode survives as long as something holds it open or mapped. The process’s memory map refers to the inode, not to the path, so the mapping remains valid and demand paging continues to read from the unlinked inode.

This is why replacing a library under a running process does not crash it, and also why the replacement has no effect on it. The process continues to execute the old code until it exits. A restart of the service is sufficient to pick up the new library; the kernel is not involved. The observable evidence is in /proc/<pid>/maps, where mappings of unlinked files are annotated as deleted, and in lsof output listing open files with a link count of zero.

The practical consequence is a machine in a mixed state that no single version describes: some processes run the old library, some the new one, and communication between them crosses an ABI boundary that nobody tested. Most of the time the boundary is compatible, because the library’s soname did not change and therefore its ABI was supposed to be stable. When it did change, the package name changed too, and the old library remains installed for the processes that need it, which is the mechanism described in dependency resolution and partial upgrades.

How ftrace redirection makes a patch live

Live patching changes the code of a running kernel without restarting it. The mechanism rests on ftrace, which can redirect the first instruction of a kernel function to a trampoline. A live patch is a kernel module that registers, for each function it replaces, a pair of addresses: the function to be replaced and the replacement. Once registered, every new call to the original enters the replacement instead.

Redirecting the entry point does not solve the hard problem, which is tasks already inside the function. A task that is executing instruction 40 of the old function cannot be moved to instruction 40 of the new one, because the two functions need not have the same layout, the same local variables, or the same stack frame size. The patch therefore has a consistency model: a task transitions to the new code only at a point where it is safe, which in practice means when it is not inside any function being replaced.

The kernel implements this by checking each task’s stack. When a task is about to return to userspace, or is otherwise at a quiescent point, the live patching code unwinds its kernel stack and looks for any function address that the patch replaces. If none is present, the task is marked as patched, and from then on it enters the replacement functions. If one is present, the task is left alone and checked again later. The patch becomes fully active when no task remains in the old code.

The stack check is the fragile part. Unwinding depends on the kernel’s unwinder having reliable information, which is why live patching depends on build options that generate it. A task blocked in an uninterruptible sleep inside a function being patched does not reach a check point, so the patch can remain partially applied for as long as that task sleeps. A task that entered the kernel through assembly with an unreliable frame can block the transition indefinitely. The kernel reports these conditions rather than forcing the issue, because forcing it would mean patching a stack that is mid-execution.

The infrastructure was merged in Linux 4.0, combining the kGraft and kpatch approaches. Patches are cumulative: each one is built against the kernel version plus all previously applied patches, so they must be applied in order and cannot be selectively removed. A patch module is tied to the exact kernel build it was compiled for, in the same way an ordinary module is, which means a live patch for a kernel that has already been superseded is of no use.

The functions a live patch cannot reach

Live patching replaces function bodies. It does not change data. A patch cannot add a field to a structure, change a lock’s type, or alter the layout of anything the old and new code both touch, because tasks in the old code and tasks in the new code would disagree about the memory layout. This rules out a large class of real security fixes, including many that add a bounds check whose state must be stored somewhere.

It also cannot patch code that has been inlined. If a function was inlined into forty call sites at compile time, there is no single entry point to redirect, and the patch’s replacement is never called from those sites. The same applies to functions marked as not traceable, and to code that has already been discarded, such as the initialization sections freed after boot.

Code that runs before the patch is applied is unaffected. A vulnerability in early boot, in the decompressor, or in a driver’s probe path that has already executed remains exploited or unexploited exactly as before; the patch prevents future executions of the fixed function, not past ones. If the vulnerability allowed a privilege escalation that has already happened, the patch does not undo it.

Finally, the patch does not change what the kernel is. The running kernel’s release string, its configuration, and its loaded modules are unchanged. Anything that inspects the kernel version to decide whether a fix is present will draw the wrong conclusion, which is why live patching systems maintain their own record of applied patches rather than relying on the version.

What deferring the restart actually costs

Live patching converts a reboot into a scheduling decision, which is its purpose: a reboot is a brief outage for every service on the machine, and some machines cannot afford one on the attacker’s timetable. The cost is a machine whose kernel is a composite of its original image and a stack of patches, none of which has been exercised through a boot.

That last point is the one that accumulates. The boot path is not exercised by a live patch: the initramfs is not rebuilt and tested, the new kernel’s driver set is not probed against the actual hardware, and the new module signing configuration is not validated. A machine that defers restarts for a year has a year of untested boot configuration waiting for the first power failure, which will apply it without warning. The reboot that was deferred is not avoided; it is moved to a moment when nobody is watching.

The honest way to read the flag is therefore as a statement about the running kernel and the running processes, not about the disk. It says that the code in memory is not the code in the package database. Whether that difference matters depends on which of the three conditions set it, and the flag alone does not say.

Restoring a machine to a known state after a failed upgrade depends on having a state to restore, which is the subject of crash-consistent and application-consistent backups.