Live Patching a Running Kernel: The Consistency Problem

Replacing a function in a running kernel is the same five-byte jump described in dynamic tracing by rewriting instructions. What makes live patching hard is not the patch. It is that a thread may be inside the function being replaced at the moment of the patch, and the replacement has a different stack frame, different local variables, and a different idea of where it is. A thread that entered the old function and returns through the new one executes a mixture of two versions, and the mixture is undefined.

The redirection itself

A live patch is a kernel module containing replacement functions and a table mapping each original function to its replacement. On load, the patch verifies that every function it names is present, computes the patch site for each, and checks that the site is a suitable five-byte location; this is the same requirement ftrace has, for the same reason. Functions that were inlined into their callers, or that the compiler split into pieces, have no single site and cannot be patched; the module refuses to load rather than silently patching a subset.

The redirection then goes through the same text-patching primitive, which serialises against other patchers and synchronises the instruction caches on every CPU. After the patch, a call to the old function’s entry arrives at the new function, because the redirect is installed at the entry itself rather than at each call site. Callers that had already resolved the address (a function pointer stored in a structure, a callback registered at boot) therefore reach the replacement as well; what does not follow the patch is code that compares or records the captured address, or a call that the compiler inlined, since an inlined copy of the body has no entry to redirect.

The old code is not freed while the patch is applied. It remains mapped, because threads may still be executing it, and the patch module keeps a reference to it for as long as the patch is enabled. Only after the kernel can establish that no thread is inside the old function may the code be considered inert, and even then, the memory is not usually reclaimed, because a subsequent patch or an unpatch may need the original bytes.

What “no thread inside” means

The naive test is a reference count: increment on entry, decrement on exit, wait for zero. That requires instrumenting the function, which is not possible for a function that was not compiled with instrumentation, and it changes the function’s own behaviour and cost. It also does not compose: a function called from an interrupt handler may be entered while the counter is being read.

The kernel’s approach is to inspect every thread’s stack. Each task’s kernel stack is a known range of memory, and the patch machinery can walk it (using the unwinder described in stack unwinding without frame pointers), looking for a return address that falls inside the old function’s text. If any thread has such an address on its stack, that thread is either inside the function or was called from it, and either way it is not safe to consider the old function inactive.

The stack check is a sampling argument, not a proof. A thread could be about to call the function but not yet have the return address on its stack; the check would find nothing and the thread would then enter the old function after the patch. This race is handled by ordering: the patch is applied to the text first, so a thread that has not yet entered takes the new path when it does. The stack check is looking for threads that entered before the patch, and those have the return address on their stack by construction.

A thread can also be inside the function without being on a stack that is visible to the check: a thread blocked in a call that never returns, a thread in a signal handler that has switched stacks, a CPU executing in a context the walker cannot follow. The check covers the tasks the kernel knows about; it cannot cover code that has left the kernel’s bookkeeping behind.

Quiescence and the consistency model

The condition the patch waits for is called quiescence: every thread has, at some point since the patch was applied, left the kernel or called the patched function again, which, after the patch, means it entered the new version. A thread that entered the old function before the patch and has not returned is non-quiescent, and the patch stays in a transitional state until it becomes quiescent.

Two things can force a thread to become quiescent. The first is that it returns from the function, which it will do unless it is blocked forever. The second is that it leaves the kernel entirely: a task that returns to userspace has, by definition, unwound its kernel stack, so it cannot still be inside the old function. The second is what makes the model tractable: a patch does not have to wait for every thread to return from the patched function, only for every thread to pass through a point where its kernel stack is empty.

That is also why a task sleeping in an uninterruptible wait can stall a patch indefinitely. It is not running the patched function while it sleeps, but its kernel stack still holds the frames it was executing when it blocked, and if one of those frames is the old function, the thread never becomes quiescent. The usual resolution is to wake it, by sending a signal or closing the file descriptor it is waiting on, so that it returns through the function and finishes. Where that is not possible, the transition simply does not complete: the patch is enabled and new calls take the replacement, but the kernel keeps reporting the transition as in progress, and the operator decides whether to accept that state or to force completion.

#include <linux/livepatch.h>

static int patched_open(struct inode *inode, struct file *file)
{
	/* Replacement body; the original symbol is retained by the patch. */
	return 0;
}

static struct klp_func funcs[] = {
	{
		.old_name = "device_open",
		.new_func = patched_open,
	},
	{ }
};

static struct klp_object objs[] = {
	{ .funcs = funcs },   /* vmlinux */
	{ }
};

static struct klp_patch patch = {
	.mod = THIS_MODULE,
	.objs = objs,
};

Stacks are not the only state

A function’s frame is not the only thing that carries its version. A structure allocated by the old function and freed by the new one, a global variable whose meaning changed, a lock taken in one version and released in the other. None of these appear in a stack walk, and all of them can produce a crash that the consistency model did not prevent.

The practical rule is that a patch may change the implementation of a function only if the data it touches keeps the same layout and the same invariants. A patch that adds a field to a structure is not a live patch; it is a new kernel. The mechanism does support replacing a function with one that calls the original, and it supports disabling a patch again, but neither changes the fact that the data structures are shared between versions and must be compatible.

The other state that matters is the per-CPU state. A function that runs with preemption disabled on one CPU, or in an interrupt handler, has a stack that the walker may not see from the context doing the check. The patch machinery handles this by checking from a context that can see all CPUs (it runs the check with interrupts disabled on the checking CPU and inspects the others), but a CPU that is in a stop-the-world state, or that has been offline, needs separate handling. A patch applied while a CPU is offline must still be consistent when that CPU comes back, which means the patch’s own initialisation must run on that CPU before it executes any patched code.

Unpatching and the reverse problem

Disabling a patch restores the original bytes at the patch site, which means threads may now be inside the replacement function while the text is being restored. The same quiescence check applies, with the roles reversed: the patch waits until no thread is inside the replacement before restoring the original. This is why a patch that is disabled and then re-enabled does not lose its state, and why the patch module cannot be unloaded until the disable has completed.

The failure mode of a forced removal is the one to keep in mind: a patch module removed while a thread is inside a replacement function leaves the thread executing code that has been freed. The kernel can refuse the removal by default; once an operator overrides that refusal, removal of patch modules is disabled for the rest of the boot, because the kernel has no way to re-establish that the code is unused. Taint arrives earlier than any of this: a kernel carrying an enabled patch is already marked, so fault reports from it describe a kernel whose code may have been replaced underneath a subsystem, and the reported offsets may not correspond to any single build.