Kexec: Booting a Second Kernel Without Firmware

A normal reboot goes through firmware. The operating system writes a reset command to the chipset, the processor resets, the firmware runs its power-on sequence, trains memory, enumerates buses, and eventually loads a boot target. That sequence takes seconds, and it destroys the contents of memory in the process: the firmware writes over low physical addresses, the memory controller is retrained, and anything the previous kernel left behind is gone.

Kexec skips the firmware. A running kernel loads a complete replacement kernel image into memory, prepares the machine state the new kernel expects, and jumps to it directly. The old kernel’s memory is not reset, its devices are not re-enumerated by firmware, and the new kernel starts within a fraction of the time a firmware boot would take. The same mechanism, pointed at a small reserved kernel, is what makes it possible to capture a dump of a crashed system.

Loading an image into a running system

The interface is a system call that takes a list of segments plus a separate entry address. Each segment describes a piece of the new kernel: a source buffer in the caller’s address space, its length, the physical destination address, and the length of the destination range. The kernel copies each source buffer into the destination physical range, which must not overlap memory the current kernel is using.

The segment list is capped at a small fixed maximum, sixteen entries, so a kernel image with many independently placed parts must be described in at most that many runs. The loader that prepares the image is responsible for arranging the pieces to fit: a typical x86-64 kernel image is a handful of segments covering the decompression stub, the compressed payload, and the parameters page.

A second interface takes a file descriptor instead of a segment list. In that form the kernel parses the executable format itself, extracts the segments, and performs the placement. The advantage is that the parsing, which is security-sensitive because it reads attacker-supplied structure, happens in kernel code that has been reviewed for it rather than in a user-space loader. The disadvantage is that the kernel must understand the image format, so the interface is architecture- and format-specific.

Before any of this can be used, the destination memory must be free. The kernel marks the ranges the caller names as reserved so that the page allocator will not hand them out, and the reservation persists for the life of the boot. A kexec load into memory the kernel later allocates over produces a second kernel whose text is silently overwritten, which fails in ways that look like a hardware fault.

The jump

Transferring control is not a single jump. The new kernel expects a machine state that the old kernel cannot produce from its own context: paging disabled or configured with the new kernel’s tables, interrupts off, the other processors parked, and the boot parameters at an address the new kernel’s entry code reads. The old kernel also cannot simply call the new kernel’s entry point, because that entry point is written for a state the calling code is not in.

The resolution is a small relocation blob, often called purgatory, that the kernel places in memory alongside the new image. It is position-independent, it is linked to run at an address it computes at runtime, and it performs the last steps: it verifies the integrity of the image it is about to jump to, sets up the registers the entry code expects, and jumps. Because it is position-independent and does not call back into the old kernel, it can run after the old kernel’s page tables have been replaced.

On x86-64 the sequence involves a control page that the kernel fills with the addresses and flags the blob needs, a switch to an identity mapping that covers both the blob and the new kernel’s physical range, and a jump to the new kernel’s entry point at its physical address. The new kernel’s own decompression stub then runs as it would after a firmware boot, except that the memory map it receives describes a machine that has already been initialized.

/* the user-visible segment descriptor; entry is a separate argument */
struct kexec_segment {
    void          *buf;    /* source in the caller's address space */
    size_t         bufsz;  /* bytes to copy */
    unsigned long  mem;    /* destination physical address */
    size_t         memsz;  /* bytes reserved at the destination */
};

The memsz field being separate from bufsz is what allows a segment to be placed in a region larger than its contents, which the new kernel may require for alignment or for a stack it will set up itself. The mem field is a physical address, not a pointer, and a caller that passes a virtual address produces a second kernel loaded into whatever physical memory that virtual address happens to map to.

Why the crash kernel needs a reservation

For a dump, the second kernel must be able to run after the first has crashed, which means its memory must be intact even though the first kernel was in an arbitrary state when it failed. That is only guaranteed if the memory was never part of the first kernel’s allocator.

The reservation is made at boot, from the kernel command line, in the form of a size and an optional offset. The kernel marks the range reserved in its memory map before the page allocator is initialized, so no page from that range is ever allocated, and no kernel data structure ever lives there. The reservation costs that much memory for the entire uptime of the system, which is the price of being able to dump after a crash.

The dump kernel is loaded into the reservation at some point during normal operation, either at boot or later when an operator requests it. When the first kernel panics, the panic path does not go through the normal reboot machinery. It jumps to the pre-loaded image, which starts with a fresh kernel that has one job: read the crashed kernel’s memory and write it somewhere.

The crashed kernel’s memory is exposed to the dump kernel as a device, conventionally a file that yields an ELF core image. The dump kernel’s user space reads that file and writes it to a disk, a network target, or a device the platform provides. The reading is possible because the dump kernel’s page tables can map the old kernel’s physical memory directly; nothing about the old kernel’s virtual mappings is preserved or needed.

Devices, DMA, and the state the firmware would have reset

Skipping firmware means skipping the hardware reset that firmware performs. Devices that were in the middle of a DMA transfer when the old kernel jumped continue to write to memory, and their writes land wherever their descriptors point. In a normal boot, firmware resets each device during enumeration and the descriptor rings are rebuilt by the new driver. Under kexec, the new kernel’s driver programs the device again, but between the jump and the driver’s initialization the old descriptors are still live.

For a normal kexec reboot this is a race that is usually won by shutting devices down before the jump. The old kernel quiesces drivers, stops DMA, and disables bus mastering where it can. For a crash dump it cannot: the old kernel crashed, its drivers are in an unknown state, and there is no opportunity to quiesce anything.

The mitigations are architectural. An IOMMU, if the platform has one and the old kernel configured it, confines a device’s writes to the address ranges its page tables permit, which limits the damage to memory the old kernel had already given that device. The dump kernel can also avoid initializing devices that might be mid-transfer, and it can restrict itself to a small set of known-safe output paths.

The deeper issue is that some hardware requires a full reset to reach a known state, and only firmware performs that reset. A device left with a partially programmed state machine may not respond to the new kernel’s initialization sequence at all. This is why kexec is reliable on platforms whose devices are well behaved after a soft reset and unreliable on platforms where they are not, and why crash dumping is sometimes configured to fall back to a firmware reboot when the dump kernel cannot start.

Limits

The reservation is a fixed cost that must be sized before it is needed. Too small, and the dump kernel cannot start or cannot map enough of the crashed kernel’s memory to be useful; too large, and the memory is unavailable to the running system for its entire uptime. The size depends on the dump kernel’s own footprint, the amount of memory that must be dumped, and the compression applied to the dump, none of which is known precisely at boot.

The second limit is that kexec does not reset the platform. Firmware settings changed by the old kernel, such as a chipset register written during initialization, persist. A new kernel that assumes the firmware’s power-on defaults will find something else. Kernel code that runs after a kexec boot is therefore written to tolerate a machine that has already been initialized once, which is a weaker assumption than a firmware boot allows and a source of bugs that only appear on the kexec path.

A third is that the mechanism is not a substitute for a watchdog. A kernel that hangs without panicking never reaches the jump, and the machine stays hung. The dump path is entered from the panic handler and from an explicit crash request, and a hang in the scheduler or in a driver bypasses both.

The image that kexec loads is the same compressed kernel described in kernel decompression and physical relocation, and it runs its own stub on entry. What it does not get is the firmware’s description of the machine, so the memory map it receives is the one the old kernel constructed, following the conventions in how firmware describes physical memory.