User-Mode Emulation and binfmt_misc for Foreign Binaries

Copy an AArch64 program to an x86-64 machine, make it executable, and run it. The kernel reads the ELF header, finds e_machine set to 183 where it expected 62, and returns ENOEXEC. The shell reports a format error and nothing else happens. Yet with the right registration in place the same command runs the program, and uname -m inside it reports aarch64 on a machine whose kernel knows nothing about AArch64 instructions. Two mechanisms combine to produce that: the kernel’s binary format handler chain, which can be taught to hand the file to an interpreter, and a user-mode emulator, which implements the guest’s instruction set and translates its system calls into the host’s.

Neither mechanism emulates a machine. There is no guest kernel, no guest page tables, and no guest scheduler. The guest program is a normal process on the host kernel, with guest registers maintained by a translator, and every system call it makes is a host system call made on its behalf. That is enough to run a compiler’s test suite and not enough to believe anything it reports about time, memory, or hardware.

Teaching the kernel to hand the file over

The kernel’s execve path tries a sequence of binary format handlers. The ELF handler rejects a file whose e_machine, EI_CLASS, or EI_DATA disagree with the running kernel’s idea of native. A script handler recognises #!. binfmt_misc is a handler that consults a user-maintained table: if the file matches an entry, the kernel builds an argument vector that starts with the registered interpreter and appends the original path.

The table lives in a pseudo-filesystem that must be mounted:

mount -t binfmt_misc binfmt_misc /proc/sys/fs/binfmt_misc
# :name:type:offset:magic:mask:interpreter:flags
# type 'M' matches magic bytes at offset, 'E' matches the file extension.
printf '%b' ':qemu-aarch64:M::\x7fELF\x02\x01\x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x02\x00\xb7\x00::/usr/bin/qemu-aarch64-static:F' \
  > /proc/sys/fs/binfmt_misc/register

The magic string above matches an ELF64 little-endian file whose type field is ET_EXEC and whose machine field is 183, which is AArch64. The mask is what makes byte-by-byte matching possible without listing every irrelevant byte: a mask of \xff compares a byte and \x00 ignores it.

The F flag matters more than it looks. Without it, the kernel resolves the interpreter path at exec time, in the calling process’s mount namespace. With it, the interpreter is opened when the entry is registered and kept as a file reference, so the binary remains launchable from a chroot or a container that does not contain the emulator. Registration is per-mount-namespace state, which is why a container runtime has to register the format inside the container, and why a system that registers it at boot does so from a service that runs before anything needs it.

Two smaller flags appear in practice. C tells the kernel to compute the new process’s credentials from the binary rather than from the interpreter, and O passes an open file descriptor to the binary as argv[1] instead of its path. Both exist for cases where the interpreter is setuid or where the path is not reachable, and both change what the guest’s argv looks like — a detail that matters when the first argument is what the program uses to decide its own name.

What the emulator does with a system call

A user-mode emulator loads the guest ELF, maps its segments at the addresses the guest expects, builds a stack with the guest’s argc, argv, envp, and auxiliary vector, and begins executing translated code. Translation is done in blocks: a run of guest instructions is decoded once, emitted as host instructions, cached, and executed. Control flow that leaves a block — a branch, a call, a return — returns to the dispatcher, which looks up or builds the next block.

A system call stops the process entirely. The guest’s instruction requests a call by number, the emulator looks the number up in a per-architecture table, converts the argument structures to the host’s layout, calls the host kernel, converts the result back, and resumes. This is where most of the emulator’s code lives, because the structures are not merely different sizes — they are different sets of fields in a different order. stat, statfs, sockaddr, msghdr, iovec, termios, and the signal frame all need a translation per guest architecture. On 32-bit guests running on a 64-bit host, the time-related structures additionally need a width translation, and mmap arguments need care because the guest expects a 32-bit address space.

The translation is not always exact. A guest system call with no host equivalent returns ENOSYS, and a guest call whose semantics changed between kernel versions may behave as the host behaves rather than as the guest’s kernel would.

The parts that are faked rather than translated

Several things a program learns about its environment are synthesised rather than obtained from the kernel.

uname is intercepted. The emulator fills in the guest’s machine name and, for some architectures, a fixed release string, because a test that branches on the machine name would otherwise take the host’s path. getauxval results are constructed from the guest’s own ELF headers. /proc/self/exe resolves to the guest binary’s path only because the emulator is the process that the kernel knows about — /proc/self/maps shows host mappings of translated code, which is why reading it from inside a guest program gives an address space that does not match the guest’s.

The dynamic loader is a guest program and runs in the guest. When the guest binary is dynamically linked, the emulator must find the guest’s ld-linux-*.so and libraries, which is what the -L option is for: it names a directory treated as the guest’s root for library resolution. Without it, the host’s loader is found first and fails immediately, because it is the wrong architecture. With the wrong root, the loader is the right architecture and the libraries are not, and the failure appears as a missing libc.so.6 from a path the loader prints.

Registering the format and pointing it at a static emulator makes the whole chain work for a dynamically linked guest only if the guest’s root is also supplied, which is why the common registration uses an emulator wrapper script or an environment variable carrying -L. Passing -E LD_LIBRARY_PATH=... sets a variable in the guest’s environment, which the guest loader then honours — the host’s LD_LIBRARY_PATH is not used, since the host loader never runs.

The fidelity boundary

Everything that depends on the host’s hardware rather than on the guest’s instructions is reported as the host’s. This has consequences that are easy to overlook when the emulator is used to run a test suite.

Timing measurements are meaningless. Translated code executes at a small fraction of native speed, translation happens on first execution of each block, and the emulator’s own bookkeeping is charged to the guest’s clock. A test that asserts a deadline rather than a result will either fail or, worse, pass for reasons unrelated to the guest.

Device access is absent. ioctl calls on real devices are not translated, /dev is the host’s, and any code path touching a framebuffer, a serial port, or a GPIO character device has nothing to talk to. Kernel modules cannot be loaded at all, which is why cross-compiling a module and testing it requires a real target or a full-system emulator rather than this one.

Memory behaviour differs where the guest’s page size differs from the host’s. An AArch64 binary whose linker was configured for 64 KiB pages has segment alignments and mmap expectations that a 4 KiB host cannot satisfy identically; the loader may still map it, but the program’s assumptions about alignment and about the granularity of protection changes no longer hold. Programs that call mprotect on sub-page ranges, or that rely on MAP_FIXED at a specific alignment, are the ones that break.

Signals are delivered through the host’s signal mechanism with the guest’s frame layout reconstructed on the guest stack. A program that inspects ucontext_t, installs an alternate signal stack, or uses sigreturn directly is exercising the most fragile part of the translation layer, and the layouts involved differ per architecture in ways that are not visible in portable source.

Process inspection is where the abstraction leaks worst. A debugger attached to the emulator sees translated host code, not guest instructions; the emulator’s own -g option exists to expose a guest-level debugging stub instead. Profilers see host symbols. Tools that read /proc/<pid>/syscall or PTRACE_GETREGS get the host’s view. Debugging a target binary is better done through the guest-level stub, as described in remote debugging through gdbserver, than by attaching to the emulator.

Where it earns its place

Emulation is the cheapest way to answer a question that requires executing target code but not measuring it. A configure script that wants to know whether snprintf returns the truncated length can be answered this way; a test suite for a parser, a hash function, or a serialisation library can run to completion. Build systems that insist on running their test programs become usable, which is the practical reason the registration is common on development machines.

What it cannot do is substitute for the target in any claim about performance, device behaviour, or kernel version. A program that passes under emulation has demonstrated that its logic is architecture-independent. It has not demonstrated that it works on the machine it was built for, because the machine it ran on was the build host wearing a different instruction set.