kexec and Crash Dumps: Capturing a Dead Kernel

A kernel that has panicked cannot write a file. The panic path runs with interrupts disabled, on one CPU, with unknown damage to data structures, and any attempt to use the filesystem risks compounding the failure into something that destroys the evidence. The mechanism that captures memory anyway is to load a second, small kernel into reserved memory during normal operation, and then, on panic, jump to it without going through the firmware. The second kernel boots with a fresh view of the hardware, reads the first kernel’s memory as plain data, and writes a dump to disk using its own drivers.

Two handovers, not one

The first handover is at load time. Userspace hands the kernel an image (a kernel binary, an initrd, a command line) and the kernel validates it and copies it into a region of memory it reserves for the purpose. Reserving means removing the pages from the allocator’s view and marking them as belonging to a crash region, so that normal operation never touches them. The region is sized by a boot parameter and must be large enough to hold the image plus its runtime allocations; too small and the second kernel cannot boot, too large and the first kernel permanently loses that memory.

The load is a system call, kexec_load, which takes the image segments, the entry point, and flags. There is a sibling, kexec_file_load, which takes file descriptors for a kernel and an initrd and lets the kernel itself do the parsing and verification. The file-based variant is what a signed-boot path uses, because the kernel can verify the image’s signature before it is ever executed; the raw variant accepts arbitrary segments, which makes it a tool for a system whose root user is trusted.

The second handover is at panic. The panic path calls into the kexec machinery, which stops the other CPUs, disables devices that would interfere, and jumps to the pre-loaded entry point. The jump is not a firmware call: it is a direct branch into the new image, which means the firmware’s own initialisation is skipped entirely. That is what makes the handover fast (no POST, no device enumeration by firmware), and it is also the source of the constraints the second kernel must satisfy, because the hardware is not in the state the firmware would have left it.

What the second kernel may not assume

A kernel written for a normal boot assumes it starts from a known state: memory in a default pattern, interrupts routed as the firmware left them, timers stopped, no other CPU executing. After a kexec handover, none of that holds. The crashed kernel’s CPUs are halted in an arbitrary state, the interrupt controller is configured for the first kernel’s drivers, the IOMMU may be translating addresses that the second kernel does not expect, and memory contains the first kernel’s data.

The second kernel therefore has a boot path of its own. It marks the crashed kernel’s memory as reserved so that its allocator never touches it, it reinitialises the interrupt controllers rather than trusting their state, and it avoids DMA until the IOMMU is reset. The reserved-memory step is the critical one: without it, the dump kernel’s own allocations would overwrite the very data the dump is meant to preserve, and the result would be a file that looks complete and describes nothing.

The memory to be preserved is described to the second kernel as an ELF core header: one program header per physical range, covering the crashed kernel’s memory from the bottom of RAM upwards and free pages included, plus a notes region holding the kernel version and the crash metadata. The address of that header is handed over on the boot command line of the second kernel, which is why the dump utility can find it without knowing anything about the crashed kernel. There is no in-use or free classification in the header itself; deciding which pages are worth writing is done afterwards by the dump utility, which reads the crashed kernel’s own bookkeeping to find out which pages were page cache, which were anonymous, and which were unused, and applies filters on that basis. Excluding genuinely free memory is an optimisation, not a correctness requirement, and it can reduce the dump to a fraction of physical memory on a machine that was mostly idle.

Where the dump goes and what it looks like

Writing the dump requires a filesystem. The dump kernel mounts one using a minimal set of drivers compiled into the initrd that was loaded alongside it, and it needs a target: a local filesystem, a raw block device, or a network protocol. The choice is constrained by what is available after a crash: a driver that needs firmware loaded from the crashed kernel’s filesystem is not usable, and a network path needs an address configuration that does not depend on a daemon that is no longer running.

The output format is ELF, for the same reason a core dump is: it is a standard description of a set of memory ranges with their addresses, and every debugger already reads it. Each range becomes a program header pointing at the bytes captured from that physical address. The crashed kernel’s virtual address mapping is not preserved (the dump describes physical memory), so a debugger needs the crashed kernel’s symbol table and its page tables to reconstruct the virtual view. The symbol table comes from the kernel binary; the page tables are in the dump, which is why the dump includes the ranges holding them.

Some formats carry additional sections for metadata: the kernel version, the panic message, the register state of the panicking CPU, and the log buffer. Those sections are what let a reader identify the crash without loading symbols first, and they are what makes a dump from an unknown machine interpretable at all.

/* The minimal description of one region: a half-open address interval
 * and a tag saying what the region was used for in the crashed kernel. */
struct dump_region {
	u64 start;   /* physical address, page aligned */
	u64 end;     /* exclusive */
	u32 type;    /* in use, free, reserved, or crash notes */
};

/* Regions must not overlap and must be sorted by start address,
 * so the dump kernel can mark them reserved with a single linear pass. */

The cost, in memory and in time

The reservation is the first cost. A dump kernel needs enough memory to hold its own image, its initrd, its page tables, and the buffers used to write the dump. On machines with little memory the reservation is a meaningful fraction; the boot parameter that sizes it is often set to a few hundred megabytes on large systems and to a much smaller value on constrained ones, and getting it wrong produces a dump kernel that boots and then fails for lack of memory.

The second cost is the write itself. The dump is a copy of physical memory, and its size is bounded by the machine’s RAM, not by what is interesting. A dump of a machine with many gigabytes produces a file of comparable size, written at whatever rate the dump device sustains, so the write occupies that device for a period proportional to the amount of memory captured. Filtering (excluding free memory, excluding ranges belonging to devices, compressing the output) reduces both, and each filter is a decision about what evidence to discard. The conservative choice captures everything and accepts the size.

The third cost is the reservation’s effect on the first kernel. Memory removed from the allocator is memory unavailable for page cache and for anonymous mappings, which is why the reservation is not enabled by default on systems that do not need it.

When the mechanism does not work

The dump path depends on the hardware still being usable. A machine check that has taken out the memory controller, or a firmware fault that has left the platform in an undefined state, means the second kernel cannot read memory or cannot write the dump. The mechanism captures software faults; it does not capture a platform that has stopped functioning.

It also depends on the jump succeeding. Secure boot configurations may refuse to execute an unsigned image, which is why the file-based load path exists and why a distribution’s kdump configuration involves a signed kernel and a matching verification policy. A kernel that is locked down may restrict the raw load path entirely, on the reasoning that replacing the running kernel with an arbitrary image is exactly the capability lockdown is meant to remove.

The interaction with the kernel log is worth stating. The console output at panic is truncated at whatever the console managed to transmit; the log buffer in memory holds more, and the dump captures it. This is the practical argument for a dump over a serial console: the serial line is a lossy, slow channel that may not be attached at all, while the buffer is the producer’s own record. A dump that includes the log buffer therefore recovers messages the console never showed, including the ones printed by the panic path itself.

The remaining gap is the moment of the crash. Memory is captured as it was when the second kernel read it, which is after the panic path ran and after the jump; anything the panic path overwrote is gone. A dump is a snapshot taken after the fact, not a recording of the fault, which is why the register state preserved in the crash notes matters, since it is the only record of the CPU’s state at the instant the fault was taken, and everything else describes what the kernel did afterwards. The crash notes are therefore read the same way as the register dump in a live fault report, with the same caveat that the saved program counter points at where execution stopped rather than at the line that corrupted state.