Decoding a Kernel Fault Report

When the kernel takes an unrecoverable fault, it prints a report and, depending on configuration, either stops or continues in a damaged state. The report is not a crash dump and not a symbolic description of a bug; it is a snapshot of machine state at the faulting instruction, formatted by the fault handler. Reading it well means separating what it proves from what it merely suggests, and knowing which fields are reliable on which architecture.

The header lines establish context

The first lines identify the kernel, the fault class, and the conditions. The kernel version string is followed by the build timestamp and the compiler version, which together identify a build well enough to match a source tree. Then comes the fault description (a page fault with the faulting address, an invalid opcode, a general protection fault with the segment selector) and the error code the processor pushed, if the architecture provides one.

On x86-64 the page fault error code is a bit field, and reading it is the highest-value step in the whole report. The low bit distinguishes a protection violation from a page-not-present fault. The next bit distinguishes a write from a read. The next says whether the access happened in user mode or kernel mode. The next indicates a reserved bit was set in a page table entry. The next says whether the processor was executing an instruction fetch. Two more bits report whether the access was a shadow-stack access and whether the fault involved a protection key.

Those bits resolve the most common ambiguity immediately. A not-present fault on a write in kernel mode at an address near zero is a null pointer dereference: the code computed a pointer from a null base and stored through it. A protection fault on a write in kernel mode at an address that belongs to a freed object is a use-after-free, because the page is still mapped but its permissions no longer allow the access. The same defect on an object whose page has been returned to the allocator and unmapped produces a not-present fault instead, which is why one bug can appear as either class depending on when the stale pointer is used. A not-present fault on an instruction fetch means execution reached an address with no code, which is usually a corrupted return address or a call through a stale function pointer. The same address can produce different reports depending on which of these it is, and the error code is what distinguishes them.

The taint flags follow the version. Each letter marks a condition that makes the kernel’s behaviour unrepresentative: a proprietary module was loaded, a previous warning occurred, a machine check happened, the kernel was forced to load or unload a module, an out-of-tree module was loaded. A report from a tainted kernel is not necessarily wrong, but the taint tells a maintainer that the code being blamed may not be the code that ran. Any diagnosis of a report with a taint flag has to account for it.

The register dump

The dump prints every general-purpose register, the stack pointer, the instruction pointer, the flags, and often the segment registers and control registers. The registers are the state at the fault, which is not the state at the call that caused it: an optimised function may have reused a register several times between the call and the fault.

The instruction pointer is the anchor. The report prints the symbol containing it and the offset within that symbol, computed by subtracting the symbol’s start address from the instruction pointer. The offset is what identifies the exact instruction, and it is only meaningful against the exact binary that produced the report. A report from a distribution kernel has offsets into a build that a developer must obtain before the offset means anything; a report from a locally built kernel can be resolved with the debug information still on the build machine.

The other registers are clues rather than facts. A register holding a small integer where a pointer is expected suggests the pointer was computed from something that was never initialised. A register holding a value that looks like an ASCII string suggests a structure was treated as the wrong type. A register holding a value equal to one of the arguments printed earlier suggests the fault is in the callee and the value arrived intact. These are pattern matches, and they are worth forming only after the faulting instruction has been read.

Reading the faulting instruction requires disassembly at the reported offset, and the useful question is which operand faulted. A load from an address in a register that is null is a different bug from a store to an address computed as base plus a large offset, because the first is a missing check and the second is usually a corrupted offset field.

The call trace

The trace below the registers is produced by the unwinder, and its reliability depends on the mechanism described in stack unwinding without frame pointers. Frames are printed from innermost to outermost, each with the symbol and offset. A trace that ends abruptly, or that contains an address that resolves to no symbol, has hit the limit of what the unwinder could establish.

The trace establishes a path, not a cause. The innermost frame is where the fault happened; the frames below it are the calls that led there. A fault inside a memory-copy routine, called from a device driver, called from a workqueue, tells a story: the driver asked to copy a buffer whose address or length was wrong. The fault is in the copy routine because that is where the bad address was dereferenced, not because the copy routine is defective.

Two patterns in traces deserve naming. A trace that ends in a function that is not expected to call anything (an interrupt entry, a scheduler path, a context switch) usually means the unwinder lost the frame rather than that execution jumped. A trace where the same function appears twice in a row with different offsets can be genuine recursion or can be a stack overflow that has overwritten the frame, and the two are distinguished by whether the stack pointer is still inside the stack.

Stack overflow has its own signature. The kernel stack is a fixed size, and on x86-64 with the common configuration it is 16 KiB, which is four pages. A fault whose address is just below the stack’s base, or a trace that is implausibly deep with repeating frames, points at exhaustion. The guard page below the stack turns the overflow into a fault rather than silent corruption, which is why the report exists at all; without the guard, the overwritten memory would be some other task’s data and the failure would appear somewhere unrelated.

The module list and the code bytes

The report ends with a list of the loaded modules, by name, each annotated with a taint and an initialisation marker. This is not decoration. Every symbol in the trace must be checked against this list, because a symbol that belongs to a module can only be resolved against that module’s own binary, and a trace naming a function from a module that is not in the list is a trace from a different boot or one whose module list was truncated. The list carries names, not load addresses; the base address needed to resolve an offset comes from the module table the kernel exposes at runtime, and only agrees with the boot that produced the report if the module was loaded at the same place, which is itself an assumption that the report cannot confirm.

Some reports also print a hex dump of the bytes around the faulting instruction. That dump is the most direct evidence available: it shows what the processor was actually executing, which may differ from what the source says if the text was patched. A dump containing a breakpoint byte or an unexpected jump where a call should be indicates that instrumentation was active at the fault, which is worth knowing before concluding anything about the original code.

/* Shape of the x86-64 page-fault error code, for reading a report. */
#define PF_PRESENT   (1u << 0)   /* 0: page not present, 1: protection violation */
#define PF_WRITE     (1u << 1)   /* 1: access was a write */
#define PF_USER      (1u << 2)   /* 1: access originated in user mode */
#define PF_INSN      (1u << 4)   /* 1: fault on instruction fetch */

static const char *classify(unsigned long err)
{
	if (!(err & PF_PRESENT) && (err & PF_WRITE))
		return "write to an unmapped address";
	if ((err & PF_PRESENT) && (err & PF_WRITE))
		return "write to a read-only or protected page";
	if (!(err & PF_PRESENT) && (err & PF_INSN))
		return "execution reached unmapped memory";
	return "read of an unmapped or protected page";
}

From report to hypothesis

The productive order is: version and taint, then fault class and error code, then faulting instruction, then trace, then module list. Each step narrows the next. A report with a taint flag from a proprietary module and a trace that ends inside that module’s code is a report about that module, and the kernel’s own code is unlikely to be at fault. A report with no taint, a null write, and a trace through a well-known subsystem is a report about that subsystem.

The step that is most often skipped is confirming the binary. Offsets are only meaningful against one build, and a report whose version string does not match the source tree in hand cannot be resolved by reading code at the reported offset. The kernel’s build identifier is long enough to be unambiguous, and matching it is a precondition rather than a formality.

What the report cannot provide is the sequence of events that led to the bad state. A corrupted pointer may have been corrupted minutes earlier, by a write from an unrelated code path, and the fault is the first observation rather than the cause. Tools that record history are what turn a snapshot into a sequence: the tracing mechanisms described in dynamic tracing by rewriting instructions, or a crash dump that preserves memory for later inspection. Without them, the report supports a hypothesis about the faulting instruction and nothing more.