Cross-Compiling Out-of-Tree Kernel Modules

An out-of-tree kernel module is the one cross build where the C library is irrelevant. The module is a relocatable ELF object that the kernel’s loader links against the running kernel’s symbol table, and the symbols available to it are exactly those the kernel exports. There is no libc.so to resolve against, no crt1.o, and no dynamic loader. What replaces them is a kernel build tree, and the tree has to match the target’s kernel closely enough that structure layouts, configuration macros, and the exported symbol set all agree with the image on the device.

That requirement is stricter than it looks, and it is the reason a module compiled for a kernel that is “the same version” can still refuse to load. The build tree is not a set of headers describing an interface; it is a snapshot of one particular kernel build, including the compiler’s decisions about it.

What the module build actually consumes

The make invocation for an external module names the kernel tree and the module directory:

obj-m := sensor.o
sensor-y := sensor-core.o sensor-sysfs.o

KDIR ?= /lib/modules/$(shell uname -r)/build
PWD  := $(shell pwd)

all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules

The -C change of directory is what loads the kernel’s own makefiles, which supply the compiler flags, the include paths, and the rules. M= names the directory containing the module’s Kbuild or Makefile, and modules is the target that builds every object listed in obj-m.

For a cross build the same invocation gains two variables:

make -C /path/to/linux M="$PWD" \
     ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- modules

ARCH selects the architecture’s arch/<arch>/ include paths and its default configuration fragments. CROSS_COMPILE is a prefix, not a path to a compiler: Kbuild prepends it to gcc, ld, ar, nm, and objcopy. Getting the trailing hyphen wrong is the usual first failure, and the error is a missing command name rather than a missing file.

The tree must have been prepared. make modules_prepare generates include/generated/autoconf.h, include/config/, the architecture’s asm-offsets.h, and the bounds and time-constant files whose values are computed by compiling and running small programs during the kernel’s own build. Those generated values are why the tree cannot be replaced with the kernel’s source directory: they encode decisions — word size, HZ, alignment, structure offsets — that only a configured build has.

modules_prepare does not produce Module.symvers. That file, which lists every symbol the kernel exports together with a checksum, is written by a full kernel build. Without it, modpost warns that the module uses symbols that are not defined in the tree and produces a .ko with no version information. The module can still be useful, but the version checks described below cannot run, and a symbol that the target kernel does not export will fail at load time rather than at build time.

Two compilers in one build

Kbuild builds some of its own tools as it goes: fixdep, which rewrites dependency files; modpost, which reads the module’s undefined symbols and produces the .mod.c file that carries the metadata; and, when the configuration is regenerated, the configuration parser itself. These run on the machine performing the build, so they must be compiled with the host compiler.

The distinction survives in the variable names. HOSTCC and HOSTCXX name the host compiler; CC and CROSS_COMPILE-prefixed tools produce target code. Setting CC to the cross compiler at the top of a module Makefile, or exporting CC in a wrapper that also drives the kernel build, breaks the tools: fixdep and modpost are then compiled for the target and cannot execute. Kbuild sets these itself, and a wrapper should pass ARCH and CROSS_COMPILE rather than a compiler.

This is the three-machine model in a single build, and the reason build, host, and target machines keeps them separate. The target compiler produces the module; the host compiler produces the tool that post-processes it. A Canadian cross of the kernel itself, where the build tree was produced on yet another machine, adds a third.

What vermagic enforces

Every module carries a string called vermagic, written into a .modinfo section. It combines the kernel release with the significant configuration choices that change the module ABI: whether symmetric multiprocessing is enabled, the preemption model, whether module unloading is permitted, and the architecture name. A kernel whose configuration differs in one of those respects produces a module with a different string, and the loader refuses to insert it with a message naming both the module’s version magic and the kernel’s.

The check exists because a module compiled against an SMP kernel has different locking requirements and, more concretely, different structure layouts and inline function expansions than one compiled against a uniprocessor kernel of the same version. The release string in the vermagic comes from the tree’s include/config/kernel.release, which is also the directory name under /lib/modules. A tree configured from a defconfig that reports a different local version produces modules that will not load on the device even though the source is identical.

CONFIG_MODVERSIONS adds a second layer. With it enabled, the kernel records a checksum of each exported symbol’s type signature, and modpost writes one entry per symbol the module uses into a __versions section. At load time the loader compares each entry with the kernel’s own table and rejects a module whose checksum disagrees, with a message naming the symbol. The mechanism catches the case vermagic cannot: a kernel rebuilt from the same source with a configuration change that altered a structure passed through an exported function, while the vermagic string stayed the same.

CONFIG_MODULE_SIG adds a third. A signed kernel with signature enforcement refuses an unsigned or wrongly signed module regardless of what its symbols say. A cross build that signs modules needs the signing key that the target’s kernel trusts, and a build that strips the signature produces a file that inserts on a permissive kernel and fails on a locked-down one. The failure is deliberate and cannot be worked around from the module side.

Configuration is the contract

The most reliable way to build a loadable module is to obtain the target’s configuration rather than to approximate it. A kernel built with CONFIG_IKCONFIG_PROC publishes its configuration at /proc/config.gz on the running device; feeding that file to make olddefconfig in the source tree produces a tree whose generated headers match the image, provided the source version and the compiler family also match.

The compiler matters more than its version number. Configuration options that record the compiler — the stack protector’s chosen mode, the set of enabled warnings, CONFIG_CC_IS_GCC against CONFIG_CC_IS_CLANG — change inlined code and, in some cases, structure layout. A kernel built with Clang and a module built with GCC of a compatible version usually work, but the combination is not guaranteed, and the failure it produces is a checksum mismatch or a crash rather than a build error.

A stronger example is structure layout randomisation. A kernel configured with the layout-randomising plugin permutes the fields of structures at build time using a seed derived from the build, so the layout the running kernel uses cannot be reconstructed from the headers alone; the plugin must be enabled with the same seed when the module is built. Nothing in the module’s metadata records the seed, so a mismatch produces a module that loads and then reads every field at the wrong offset.

What the module cannot use

Kernel code is freestanding. It runs with no C library, no startup files, and no floating-point context on architectures where the kernel does not preserve one across calls. The allocation functions are the kernel’s own, the output function is the kernel’s log buffer, and the string functions come from the kernel’s own implementation rather than libc’s — which matters for a build that includes a header from the sysroot by accident, because the symbol will be resolved at load time against the kernel’s exports and fail.

Static linking against a sysroot library is equally wrong for a different reason: the module is not linked in the ordinary sense. It remains a relocatable object with undefined symbols, and the loader resolves them against the kernel’s export table. Symbols the kernel does not export are undefined at load time even if they exist somewhere in the target’s root filesystem, and the loader’s error names each one.

The stack is small by user-space standards, measured in kilobytes and shared with interrupt handling on some configurations. Deep recursion that would be acceptable in a user program overflows it, and the resulting corruption is not attributed to the module by any diagnostic. Floating-point arithmetic in a kernel module is similarly unsupported on many architectures, and the compiler does not prevent it unless the arch’s build flags say so.

Diagnosing a module that builds and will not load

modinfo reads the metadata without loading anything: the vermagic string, the dependencies, the source version, and any parameters. Comparing its vermagic output with the kernel’s own string — visible in the loader’s error message and in /proc/version — distinguishes a configuration mismatch from a symbol problem.

nm -u module.ko lists the undefined symbols the loader will try to resolve. Comparing that list with the kernel’s export table identifies the missing ones before the module is copied to the device. A version mismatch produces a loader message naming the symbol and both checksums, which points at a configuration difference rather than at a typo.

The build itself is the last place to look. A module that compiles with warnings about implicit declarations is usually including a header from the wrong tree, and a module whose .ko is produced without modpost’s symbol warnings is usually building in a tree that never had a full build. Both are visible in the build output, and both produce a file that exists, is the right architecture, and is rejected by the loader for reasons that have nothing to do with the code.