Anatomy of a Cross Toolchain: Triples, Sysroots, and Multilibs

A native compiler is allowed to make assumptions that a cross compiler cannot: that the C library exists, that its version matches the headers it was built against, that the program is loaded by a dynamic linker at an address the kernel chooses, and that a main function will be called with an argument vector prepared by something else. Firmware has none of these. The program is the only thing running, it is entered at a vector table, and its C library has to be supplied with the image rather than found on the machine.

A cross toolchain is therefore not a different compiler. It is the same compiler with a different set of assumptions baked in at build time, plus a runtime library written for a machine that has no operating system.

The triple selects an ABI, not a manufacturer

The string arm-none-eabi looks like a vendor description but is a four-field ABI selector: architecture, vendor, operating system, and ABI or environment. The vendor field is conventionally none for bare metal and carries no meaning to the tools; the operating system field is what actually changes code generation. arm-none-eabi produces objects for a system with no loader and no dynamic linking, which means position-independent code is off by default, the default linker script is a bare-metal one, and the C library selected is a bare-metal implementation.

Change the vendor and system fields and the same architecture produces incompatible output. arm-linux-gnueabihf links against glibc, emits calls into a dynamic loader, expects a kernel to service system calls, and marks its objects as using the hard-float variant of the ABI. Objects from the two toolchains share an instruction set and cannot be linked together: the unresolved symbols, the startup objects, the calling convention for floating-point arguments, and the expected presence of a PLT all differ.

The environment field matters most for floating point. On 32-bit Arm the same instruction set supports three distinct conventions for passing a float or double: everything in general-purpose registers, floating-point arguments in floating-point registers with a software fallback, and floating-point arguments strictly in floating-point registers. These are separate ABIs with separate library builds, and the compiler records its choice in a note section so the linker can refuse to combine incompatible objects. The check is not advisory. A linker that sees conflicting floating-point attributes in .ARM.attributes will stop with an error rather than produce an image that crashes on the first call across the boundary.

What the sysroot has to contain

The compiler’s include search path and library search path are rooted at a sysroot: a directory tree holding the target’s headers and libraries in the layout the target would have them. --sysroot= repoints every absolute path the compiler would otherwise resolve against the build host. That single flag is what prevents the most common cross-compilation accident, where a build picks up the host’s /usr/include/stdio.h and compiles against a libc the target will never run.

A sysroot for a bare-metal target typically holds target-specific headers under usr/include, startup objects and the C library under usr/lib, and one library set per multilib variant. The startup objects deserve attention because they are where the C abstract machine meets the reset vector. A hosted C implementation calls main from _start after the loader has set up the environment; a bare-metal one calls main from a startup file that has already copied initialised data into RAM, zeroed the block starting symbol region, and possibly enabled a floating-point unit. Which of those steps happens, and in what order, is fixed by the startup object and the linker script, not by the C standard.

Freestanding compilation is the mode that acknowledges this. Under -ffreestanding the compiler stops assuming that main has a special form, that the program starts at main, and that the standard library is present. It still emits calls to memcpy, memset, and memmove for aggregate copies and array initialisation, which is the detail that catches builds made with -nostdlib and then found to contain an undefined reference to memcpy in code that calls no library function. The routines have to come from somewhere: either the C library the sysroot provides, or a small set written for the target. A structure assignment is a copy, and the compiler is entitled to implement it with a call.

Multilib: one toolchain, several ABIs

A single installation usually ships several prebuilt library variants, selected by the code-generation flags. On Arm these are the combinations of instruction set (Arm or Thumb), architecture revision, floating-point unit, and floating-point ABI. The compiler picks the variant whose flags are compatible with the ones in effect, which means -mthumb -mcpu=cortex-m4 -mfloat-abi=hard -mfpu=fpv4-sp-d16 and the same command without -mfloat-abi=hard resolve to different libc.a files and different libgcc.a files.

The selection is not a preference. libgcc contains the routines the compiler calls when an operation has no single instruction: 64-bit division on a core without a divide instruction, soft-float arithmetic when the ABI is soft, and the unwinder support used by exception handling. Choosing the hard-float variant of libgcc for a build compiled with the soft-float ABI produces calls whose register assignments do not match, and the failure appears as wrong numeric results rather than as a link error.

Listing what an installation actually provides is a one-line query, and doing it early avoids an afternoon of confusion:

$ arm-none-eabi-gcc -print-multi-lib

Each line of the output is a directory relative to the library root, a semicolon, and an @-separated list of the flags that select it. A build whose flags match no line falls back to the default entry, which is how a project ends up silently linking a generic library while believing it selected the floating-point one. The link map shows which archive member was pulled in; reading it is the only reliable confirmation.

Failure modes that survive to the target

The expensive mistakes are the ones the tools accept. Mixing objects built with different structure packing conventions links cleanly and misreads every field past the first. Mixing objects built for different architecture revisions links cleanly if no instruction newer than the older revision appears in a shared path, and faults only when one does. Mixing newlib and a second C library links cleanly when the symbol sets do not overlap, and then produces two independent stdout states and two heaps.

The ABI also fixes details that look like compiler bugs when they are violated. The 32-bit Arm procedure call standard requires the stack pointer to be eight-byte aligned at every public interface, and requires a 64-bit value to be eight-byte aligned in memory. Hand-written assembly that pushes a single register and calls C violates the first requirement; a structure with a double and a uint32_t packed by hand violates the second. Neither produces a diagnostic, because the compiler has no way to see the assembly and the linker does not check data layout.

Alignment of the image itself is a related trap. Vector tables on Arm Cortex-M parts must be aligned to their own size in bytes rounded up to the next power of two, with a minimum of 128 bytes, and the linker places the table according to a script rather than a compiler setting. A table placed at an address that is not sufficiently aligned boots into a fault that looks like a corrupt image. The placement rules are the subject of linker script section placement, and the resulting image is best inspected as an object file rather than trusted, as described in reading ELF sections and debug information.

What the build must record

Because none of these properties are visible in a compiled binary, a build has to record them. The target triple, the architecture and floating-point flags, the library variant, the sysroot path, and the linker script all participate in determining whether two objects may be combined, and all of them can drift between a developer’s machine and a build server without any file changing.

Recording them has a second use: the same list of properties is what a debugger needs in order to interpret the image. A debug probe loads a symbol file, not the binary, and the symbol file’s addresses are only meaningful for the exact link that produced the flash contents. That correspondence, and the DWARF data that carries it, is what makes the difference between a debugger that shows source lines and one that shows a disassembly of the wrong function.