Cross-Language Calls and the Structure-Passing Contract

A foreign function interface is usually introduced as a way to call a function from another language, and the function name is the part that gets the attention. The name is the smallest part of the problem. Two languages that agree on a symbol still have to agree on how a structure is passed, who frees a returned buffer, how an error is signalled, and what happens when a callback unwinds through a frame the other language does not understand. Each of those is an ABI question, and each has a failure mode that produces corruption rather than a diagnostic.

The C ABI is the common denominator because it is the only interface that most languages implement natively. Everything below concerns what survives the translation into that denominator and what does not.

Names are the least of it, but they are not nothing

A C function compiles to a symbol whose name is the identifier, optionally with a leading underscore on platforms whose object format expects one. A C++ function compiles to a mangled name that encodes the namespace, the class, the parameter types, and the cv-qualifiers, so that overloads and templates can coexist in one symbol table. The Itanium C++ ABI defines the mangling scheme used by GCC, Clang, and most Unix toolchains; MSVC uses a different scheme. The two are not interoperable, which is why a C++ library exposed to another language is conventionally wrapped in a C interface with extern "C" linkage.

extern "C" does two things: it suppresses mangling, giving the function a plain C symbol, and it disables C++ features that the C calling convention cannot express, such as overloading and default arguments. What it does not do is change the calling convention, the structure layout rules, or the exception behaviour. A C++ function declared extern "C" still uses the C++ ABI’s parameter-passing rules, which are specified separately from the C rules and are documented to agree with them for plain aggregates on the major platforms rather than being identical by definition.

The same problem appears from the other side. A language that generates a symbol with a version suffix, a leading underscore, or a stdcall decoration produces a name the linker cannot resolve against a plain C declaration. The error is an undefined symbol at link time, which is the benign case: it fails loudly and names the symbol. The dangerous cases are the ones where the name resolves and the types do not.

Aggregates by value: where the two sides diverge

A function returning a structure by value has no single answer in the C ABI, and each target’s convention has to be implemented independently by every language that calls it.

On x86-64 System V, a structure of up to 16 bytes is returned in registers, with each eightbyte following its own class: two floating-point eightbytes come back in xmm0 and xmm1, two integer eightbytes in rax and rdx, and a mixed aggregate in whichever pair of those registers matches its per-eightbyte classification. A larger structure is returned through a hidden pointer that the caller supplies, and on x86-64 that pointer arrives in rdi, which shifts every real argument by one register. On AArch64, the indirect result register is x8 rather than the first argument register, so the arguments do not shift, but the caller must still allocate the destination and pass its address.

A binding generator that gets this wrong produces a call that reads the result from the wrong register or writes it to the wrong address. The failure is a structure full of plausible-looking garbage, not a crash, because the address in rdi or x8 may happen to be valid. The eightbyte classification that determines which of these cases applies is described in the register assignment article, and it depends on member types in a way that a naive size-based rule does not capture.

The reliable design for a cross-language interface is to avoid by-value aggregates entirely. Passing a pointer and a length, or a pointer to a caller-allocated structure that the callee fills in, removes the classification problem and makes the ownership of the memory explicit. The cost is an indirection and, for small structures, a measurable loss of efficiency; the benefit is an interface whose behaviour does not change with the target.

/* Opaque and explicit: no by-value aggregate crosses the boundary. */
struct session;

struct session *session_open(const char *path);
int   session_read(struct session *s, void *buf, unsigned long len);
void  session_close(struct session *s);

An opaque pointer plus accessor functions is the pattern that survives every language and every target. It also gives the library freedom to change the structure’s layout without breaking callers, which is the same argument made for opaque handles in linkage and symbol visibility.

Callbacks, unwinding, and the frames between

A callback is the interface running in the other direction, and it introduces a frame from one language in the middle of a call chain from another. The mechanism works because a function pointer is just an address and the call goes through the same convention on both sides. The problems are in what the callback may do.

The first is unwinding. An exception thrown in a C++ callback that propagates through a C frame compiled without unwind tables cannot be handled, and the runtime calls the terminate handler. The C++ ABI requires every frame between the throw and the handler to have unwind information, which a C compiler emits by default on the major platforms but which a hand-written assembly frame or a -fno-exceptions build may not. The reverse direction has the same problem: a longjmp out of a C callback into a C++ frame skips destructors, because longjmp restores the stack pointer and registers without running cleanup code. The stack unwinding machinery described in the call frame information article is what makes the difference between a controlled propagation and a skipped destructor.

The second is the calling convention of the callback itself. On 32-bit Windows, __stdcall and __cdecl differ in who pops the arguments, and a callback declared with one and called through the other corrupts the stack on return. On the 64-bit conventions the distinction disappears, but the declaration must still match in parameter types, and the same promotion rules apply to a callback declared without a prototype.

The third is the lifetime of the callback’s context. A callback that carries a void *user_data pointer requires the caller to keep that object alive for as long as the callback may fire. A language runtime that moves its objects, or that collects garbage, must pin the object or register a root; a runtime that does not, and passes a pointer to a stack local, produces a callback that fires after the frame is gone.

Errors, ownership, and the conventions that do not cross

C has no exceptions, so every C interface invents an error convention. The three common ones are an integer return code with an out-parameter for the result, a sentinel value such as a null pointer or -1 with a separate function to retrieve details, and a thread-local error variable read after the call. Each maps differently onto the caller’s language, and the mapping is where errors get lost.

A return code that the caller ignores is the most common defect at this boundary. A binding that translates a C function returning an error code into a function returning a value must decide what to do with the code, and the choice is not free: raising an exception is idiomatic in some languages and unavailable in others, and returning a tuple changes every call site.

Ownership is the sharper problem because it has no syntactic trace at all. A function that returns a char * may be returning a pointer to a static buffer, to memory the caller must free with a specific deallocator, or to memory owned by an object the caller must keep alive. The C declaration says none of this. A caller that frees a static buffer corrupts the heap; a caller that does not free an owned buffer leaks it. The convention has to be documented, and the only dependable enforcement is to make the allocation and deallocation functions part of the interface so that the caller never calls the language’s own free on memory the library allocated. On Windows this is not a stylistic preference: memory allocated in one C runtime and freed in another is a genuine mismatch when the two runtimes have separate heaps.

Strings carry the same ambiguity in a smaller space. A char * may be null-terminated, may be paired with a length, and may be encoded in UTF-8, in a locale-specific 8-bit encoding, or in UTF-16 with a two-byte unit. Passing a UTF-8 string to a function expecting the platform’s narrow encoding produces mojibake rather than an error, and the length in bytes differs from the length in characters for any non-ASCII input.

Testing the contract instead of assuming it

The practical defence is to make the assumptions executable. Three techniques cover most of the ground.

Static assertions on structure size and member offsets catch a layout mismatch at compile time on the side that has the header. A shared header with _Static_assert on sizeof and offsetof turns a silent disagreement into a build failure, and it is the cheapest form of ABI test available.

A version field in any structure that crosses the boundary allows a runtime check. The callee reads the version the caller wrote, rejects a version it does not understand, and the failure becomes an error return rather than a corrupted field. The version field costs four bytes and one comparison, and it is the only mechanism that catches a mismatch between two independently built components where the headers were not shared.

A round-trip test through the actual boundary exercises the calling convention rather than the declarations. A test that passes a structure by value, receives one back, and compares every field catches the register-classification error that a compile-time assertion cannot, because the two sides compile separately and both are internally consistent. The test is only meaningful when it runs on each target, since the convention differs between them.

The remaining gap is the one no test can close: a declaration that agrees with the definition at compile time and disagrees at run time, because the library was built from a different version of the header. The version check is the only mechanism that reaches that case, which is why stable interfaces carry one.