ABI Boundaries as Module Seams in Dynamic Linking

A module boundary enforced by the compiler is checked once, at build time, against declarations that both sides share. A module boundary enforced by the dynamic loader is checked at load time, against machine-level conventions that neither side can see in source form. The second kind of boundary is what makes independently versioned plugins, driver modules, and language runtimes possible, and it is also why a library that adds a private data member can crash a program that was never recompiled.

The ABI is the set of rules that lets one compiled artifact call into another: how arguments are passed, how return values are delivered, how structures are laid out in memory, how names are encoded in the symbol table, how exceptions propagate, and how the loader resolves references. It is not a document most programmers read, and it governs the behaviour of every shared library boundary in a running process.

What the ABI actually fixes

On the System V AMD64 convention used by Linux and most Unix-like systems on x86-64, the first six integer or pointer arguments travel in registers; floating-point arguments use the vector registers; further arguments go on the stack. The return value comes back in a register pair for small structures and via a hidden pointer for large ones. Which arguments are assigned to which register class, and in what order, is part of the ABI, and so is the definition of “large”.

Structure layout adds alignment and padding rules. A struct containing a char followed by an int occupies eight bytes on a typical 64-bit platform, not five, because the int must be aligned to four bytes. Adding a field at the end changes sizeof; inserting one in the middle changes every subsequent offset. A caller compiled against the old layout writes to offsets that now hold different fields, and the failure is memory corruption rather than a type error.

C++ adds name mangling and object layout. A member function’s symbol encodes its class, parameter types, and constness, so an overload set produces distinct symbols; changing a parameter type changes the symbol and the old one disappears. Virtual dispatch depends on the vtable, whose entry order is determined by declaration order. Adding a virtual function in the base class shifts the entries of every function declared after it, so a subclass compiled against the old header dispatches to the wrong slot. Adding a non-virtual member changes the size of the object, so a caller that allocated the old size now under-allocates.

None of these are defects in the language. They are the consequences of compiling two halves of a program at different times, and they are the reason an ABI-stable library must be designed around them rather than discovered by them.

The loader seam

Dynamic linking is a substitution mechanism. A reference to a function in a shared library compiles to a call through the procedure linkage table, which on first use resolves the symbol and patches the table slot to the real address, or resolves eagerly when the library is loaded with LD_BIND_NOW or built with -Wl,-z,now. The indirection is one extra jump per call after binding, and it is the price of the substitution: the address is not fixed until the loader chooses a library.

That chooser is a genuine module seam. LD_PRELOAD inserts a library ahead of the normal search order, so its definitions of exported symbols win. Test harnesses use this to replace a clock, a network client, or a file system with a substitute without recompiling the code under test. Plugin hosts use the same mechanism through dlopen and a registration function, and driver architectures rely on the loader’s symbol resolution to bind a module to a kernel interface at load time.

The seam has a sharp edge. Symbol resolution in ELF is global by default: unless a library’s internal references are bound locally, they can be interposed by another library that defines the same name. A library that calls its own malloc wrapper may find itself calling a different one, because the call went through the global symbol table. This is what makes LD_PRELOAD work for instrumentation, and what makes a shared library’s internal encapsulation weaker than its source-level visibility suggests.

The usual defences are to hide symbols that are not part of the public interface and to bind internal references locally. Compiling with -fvisibility=hidden and marking only the exported API as visible prevents other libraries from seeing the internals at all. A version script or an export list does the same at link time, and -Bsymbolic or -Bsymbolic-functions binds a library’s internal calls to its own definitions, at the cost of preventing legitimate interposition of those symbols. Each choice moves the boundary; none removes the trade.

Encapsulation in C: opaque handles

The C idiom for an ABI-stable module is a pointer to an incomplete type.

/* cache.h — the only header a consumer compiles against */
typedef struct cache cache_t;

cache_t *cache_new(size_t capacity);
void     cache_free(cache_t *c);
int      cache_put(cache_t *c, const char *key, const void *value, size_t len);
const void *cache_get(const cache_t *c, const char *key, size_t *len_out);

The struct’s definition lives in the library’s source and never appears in the header. Its size, field order, and internal data structures can change in any release without recompiling consumers, because no consumer knows them. The cost is an allocation behind the handle, a function call for every operation, and the loss of inlining and compile-time type checking of the fields. The benefit is a boundary that survives independent release cycles, which is the entire reason to have a shared library rather than a static one.

The same idiom in C++ is a pure-virtual interface plus a factory function that returns a pointer to the base. The vtable becomes the exported contract, and because the base class’s layout is fixed by the header that consumers include, adding a virtual function to the base is an ABI break unless the class reserves slots. This is why large C++ libraries often ship a C API and wrap it: the C ABI is small, stable, and implementable by any language with a foreign-function interface, while the C++ ABI varies by compiler, standard library, and exception model.

Versioning the boundary

An exported symbol can carry a version, as ELF’s .symver mechanism allows. A library can export parse@VERS_1 and parse@@VERS_2, where the double @ marks the default version for newly linked programs while old binaries continue to bind to VERS_1. The mechanism lets one installed library serve programs built against different releases, which is what makes a stable system ABI possible.

Versioning does not solve the layout problem. Two versions of a function can coexist, but a structure whose layout changed cannot be both old and new at the same address unless the library maintains two representations and translates between them. The common resolution is to keep the old entry point working on the old layout and add a new entry point for the new layout, which is a source-level design decision forced by a binary-level constraint.

The SONAME field records the library’s major version and is what the loader records as a dependency. Programs linked against libfoo.so.1 require that name at run time; publishing libfoo.so.2 with an incompatible ABI means both files stay installed, and the loader picks per program. The failure mode when this convention is ignored is the classic one: a program loads a newer library with the same name, and calls land on functions whose signatures have changed.

Where the seam costs performance

Crossing a shared library boundary removes the compiler’s ability to see the callee. Inlining stops at the boundary, as does interprocedural optimisation, constant propagation into the callee, and the removal of redundant loads across the call. The call itself goes through the PLT, which is an indirect jump, and on the first call it also involves the resolver.

The costs are real but bounded, and they scale with call frequency. A library function called once per request pays nothing measurable for the indirection; a library function called per pixel or per element pays for it in the inner loop. This is the same trade as the pimpl idiom, one level up: a boundary that buys independent release cycles is worth a few cycles per call in a cold path and is not worth it in a hot one. Systems that need both typically keep the hot data structures in the caller’s module and cross the boundary only at coarse-grained operations such as batch submission or buffer handoff.

Exception handling across the boundary is a separate cost and a separate compatibility requirement. The unwinder must understand the calling frame’s exception tables, which is why mixing compilers or exception models in one process can terminate the program instead of propagating. A C API that returns error codes avoids the problem by not throwing across the seam, which is the main reason C ABIs remain the interoperable choice.

What the boundary can and cannot promise

An ABI boundary promises call compatibility. It does not promise behavioural compatibility, and it does not promise that a substituted library will satisfy the caller’s expectations. A plugin that implements the exported functions but returns results in a different order, or that assumes a different ownership convention for returned buffers, will compile, load, and misbehave. The contract has to be documented and tested at the seam, exactly as the source-level interface contract has to be.

The boundary also inverts the usual direction of dependency analysis. A plugin architecture is a dependency cycle by construction: the host knows the plugin interface, and the plugin knows the host’s services. The graph is acyclic only when the interface is extracted into a module both sides depend on. This is why plugin systems are usually built around a small, stable interface library, and why the interface’s ABI stability matters more than the host’s.

At the largest scale, the ABI is what makes a module replaceable without rebuilding the world. It is also the point where physical modularity stops being a build-time convenience and becomes a runtime contract with its own failure modes. The layer above it — the function call overhead that the seam introduces — is the last piece of the cost model, and the reason a boundary is worth drawing only when something on the other side genuinely varies.