Opaque Handles and Ownership Transfer in C APIs

An interface that exposes a struct exposes its layout. Every field’s offset, every field’s size, and the struct’s total size become part of the contract, and a caller compiled against one layout will keep using it after the library has changed. Because the caller allocates the struct, since it must when the struct is a parameter or a returned value, the library cannot grow it without breaking every existing binary.

An opaque handle avoids this by not being a struct at all. The header declares an incomplete type, the library defines it privately, and callers only ever hold a pointer or an integer token. The size of the object is a private detail, so it can change between library versions without recompiling anything.

The incomplete type and what it buys

In C, a declaration like typedef struct db db; introduces a type name without defining its members. A caller can declare db *h, pass it, compare it to null, and store it. It cannot dereference it, cannot declare db by value, and cannot compute sizeof(db). Those three prohibitions are the entire benefit: they are the operations that would couple the caller to the layout.

C++ reaches the same place with a class whose members are declared in a private implementation header, and most other languages model it as a pointer to an unnameable type. The mechanism differs; the effect is identical: the caller’s compiler knows a size that is one pointer, and the library’s compiler knows the real size, and the two never need to agree.

The cost is indirection. Every field access becomes a load through the pointer, and the object can no longer live on the caller’s stack. For a large object that is called once and used many times, the indirection is negligible. For a small object used in a tight loop, it is not: a handle to a four-byte value costs a pointer chase per access and defeats the caller’s ability to keep the value in a register.

The second cost is allocation. Because the caller cannot place the object, the library must allocate it, and that allocation is the subject of allocator mismatch. The handle is therefore almost always paired with a create function and a destroy function, and the pair is what makes the ownership explicit.

Transfer, borrow, and the three questions

Every handle that crosses the boundary raises three questions, and an interface that answers fewer than three is ambiguous.

The first is whether the caller owns the handle. Ownership means the caller must eventually destroy it, and that failing to do so leaks. A factory function returns an owned handle. A lookup function usually returns a borrowed one.

The second is how long a borrowed handle remains valid. A handle borrowed from a container is valid until the container is modified, or until the container is destroyed, or until some other documented event. The distinction matters because the borrowed handle has no lifetime of its own; it is a view, and the view’s validity is a property of something else.

The third is what happens when an owned handle is destroyed while other handles derived from it still exist. If a statement handle belongs to a connection handle, destroying the connection invalidates the statement, and the caller who then calls a function on the statement is using freed memory. Interfaces that allow this must either document the invalidation or keep the parent alive until its children are released.

typedef struct conn conn;
typedef struct stmt stmt;

conn *conn_open(const char *path);        /* owned by the caller */
void  conn_close(conn *c);                /* invalidates every stmt from c */

stmt *conn_prepare(conn *c, const char *sql);  /* owned, parent must outlive it */
void  stmt_finalize(stmt *s);

/* Borrowed: valid until the next call that mutates the result set. */
const char *stmt_column_text(stmt *s, int i);

Four functions, and the ownership of each return value is different. conn_open transfers a handle; conn_prepare transfers a handle whose lifetime is bounded by the connection; stmt_column_text returns a borrowed pointer whose validity is bounded by the statement’s state. A binding that treats all three as owned will double-free the borrowed pointer, and one that treats all three as borrowed will leak both handles.

The borrowed-pointer case has a further wrinkle: the pointer may point into memory the library relocates. A library that reallocates its result buffer on the next fetch invalidates the pointer, and the caller that stored it in a language-level string without copying has a dangling reference. The interface must say whether the pointer is stable, and if it is not, the binding must copy immediately.

Integer handles and the ABA problem

Some interfaces return an integer token rather than a pointer. The token is an index into a table the library owns, or a file-descriptor-like value, or a generation-counted identifier. Integer handles are attractive because they are trivially copyable across every language and cannot be accidentally dereferenced.

They bring their own failure mode. If the library reuses slot numbers, a caller holding a stale token can operate on a different object than the one it intended, and the operation succeeds. POSIX file descriptors have exactly this property: closing a descriptor and opening another may return the same number, and a stale descriptor then refers to the new file. Libraries that care about this pair the index with a generation counter, so that a stale token fails a comparison even when the slot has been reused.

The size of the token also matters. A 32-bit token passed through a language whose integers are 64-bit is fine; the reverse truncates. A token that is a pointer cast to an integer must be cast back through a type of the same width, and a language that stores it in a signed 32-bit field will mangle addresses above the sign boundary on 64-bit platforms. The type must be fixed by the interface as a specific-width integer, not a language’s native int.

Pinning and the handles that hide a moving object

A handle that refers to a managed object adds a lifetime problem the C side cannot see. The object may move, so a raw pointer to it is valid only while it is pinned, and pinning has a cost: the collector cannot compact the region containing it, and a long-lived pin fragments the heap.

The handle indirection solves this by keeping the object reachable through a table that the collector scans. The table entry holds a managed reference, which the collector updates when it moves the object; the handle the C side sees is the table index, which does not change. Calling into the library with that index dereferences the current object. Nothing is pinned, so the collector is free to move, and the C side never holds an address into the managed heap.

The cost is a lookup per access and a table that must be kept in sync. The benefit is that a long-lived handle such as an event subscription, a registered callback, or a database connection wrapper does not pin an object for the life of the process.

Where pinning is unavoidable is where the C library needs the address itself and will hold it: a buffer handed to a device, a structure registered with a kernel interface, a memory region mapped for direct I/O. Those cases are covered in the discussion of zero-copy paths in string encoding and buffer ownership, and the design rule is the same: pin for the shortest interval the API permits, and unpin in a scope that cannot be skipped.

Failure modes at the boundary

Handles fail in four characteristic ways, and each has a distinct signature.

A use-after-destroy reads freed memory. The allocator may have reused the block, so the object’s fields contain another object’s data, and the operation proceeds with nonsense. Debug allocators that poison freed memory turn this into a visible failure; without them it is intermittent.

A double destroy pushes a block onto a free list twice, which corrupts the list and produces a failure at an unrelated allocation. Some allocators detect the second free and abort; most do not.

A leak of an owned handle is silent until memory pressure. The handle table grows, the library’s internal state grows, and the process eventually fails somewhere unrelated. Leak detection for handle-based APIs is best done by counting creates against destroys in a test harness, because the handles are opaque and no tool can trace them generically.

A handle used after the library has been unloaded is the sharpest. The table is gone, the code that interprets the token is unmapped, and the call faults immediately. This is why an interface that returns handles must be paired with a rule about when the library may be unloaded: only after every handle is destroyed, which in practice means the library stays loaded for the life of the process.

The consequence is that opaque handles move the coupling from layout to protocol. The caller no longer needs to know the object’s shape, but it must know the ownership of every value it receives, the lifetime of every borrowed view, and the invalidation rule for every parent-child relation. Those are the things a stable C ABI must state, because they cannot be inferred from the declaration and cannot be checked by the compiler. The advantage over exposing structs is that the protocol can be documented once and kept stable while the layout changes freely, which is precisely the property that lets one library be used from many languages over many years.