Build Identifiers and Separate Debug Information

Debug information is bulky. A C++ binary built with full debug information can be several times the size of its stripped counterpart, and the symbols describe internal structure that a deployed program has no use for. The standard resolution is to split the two: ship a stripped executable, archive the debug information separately, and give both a shared identity so a debugger can pair them later. That identity is the build identifier, and the pairing machinery is what makes a crash report from a stripped production binary resolvable to source lines.

The design constraint is that the identity must survive everything the toolchain does between compile and crash, must be cheap to compute at link time, and must not depend on file names or paths, because those change constantly.

A hash over the output, not over the source

A build identifier is computed from the contents of the linked output rather than from the sources or the command line. GNU linkers compute it after layout, over the bytes of the output with the note section itself excluded, and store the result in a .note.gnu.build-id section. The default algorithm produces 160 bits, displayed as 40 hexadecimal characters, which is the same width as SHA-1; the linker can also be asked for a smaller or larger identifier or for a UUID-style value. The section has the ordinary note structure: a name of GNU, a type of NT_GNU_BUILD_ID, and the identifier bytes as the descriptor.

Hashing the output rather than the inputs has a consequence that matters in practice: two builds from identical sources with identical flags produce the same identifier only if every byte of the output matches. Timestamps embedded by the compiler, differing temporary paths recorded in debug information, and randomised layout all change the hash. That is a feature, because it means an identifier match is evidence that the exact bytes match, but it also means that rebuilding does not reproduce the identifier, which is why the archive of debug files is organised by identifier rather than by version.

The stripped binary keeps its note. Stripping removes sections and symbols, not notes that are marked allocated, so .note.gnu.build-id remains in the shipped executable and in every core file’s mapping of it. That is the whole point: the identifier is the only link between the artifact that crashed and the debug information stored elsewhere.

Splitting and the debuglink back-reference

Two mechanisms connect a stripped file to its debug information, and they solve different problems.

The first is a section named .gnu_debuglink. It contains a file name — conventionally the original file name with a .debug suffix — and a CRC32 checksum of the debug file’s contents. A debugger that finds this section searches a list of directories for a file of that name, verifies the checksum, and loads it if it matches. The search path includes the directory of the executable, a .debug subdirectory of it, and a global debug directory, all configurable. The CRC32 is a weak check by cryptographic standards, but it is enough to catch the common failure of picking up a stale file with the same name.

The second is the build identifier path. If the debug information was installed into the global directory in the layout the identifier implies — a directory named by the first two hex characters, containing a file named by the remaining thirty-eight with a .debug suffix — a debugger can find it directly from the note without any debuglink section at all. This layout is what distributions use, because it is independent of the original file name and therefore survives renaming, and because it scales to a directory with millions of entries through a two-level fan-out.

Splitting is done by copying the debug sections out with objcopy --only-keep-debug, stripping the original, and then adding the debuglink section with objcopy --add-gnu-debuglink, which recomputes the CRC over the file it is given. The order matters: the debuglink checksum must describe the file that will be installed, so the copy must be final before the section is added.

objcopy --only-keep-debug prog prog.debug
strip --strip-debug --strip-unneeded prog
objcopy --add-gnu-debuglink=prog.debug prog
readelf -n prog | grep -A1 'Build ID'

A subtlety appears with separate debug files and relocations. The debug file preserves section addresses as they were in the linked output, so a debugger must apply the same load bias it applies to the stripped binary. For a position-independent executable the bias is whatever the loader chose at run time; the debug file contributes no addresses of its own beyond the ones already fixed at link time. A debugger that treats the debug file as a separate object with its own base produces addresses offset by the bias, and the usual symptom is a backtrace whose line numbers are plausible but consistently wrong.

Fetching over the network

Keeping a local archive only works when the crash and the symbols are on the same machine or the same administrative domain. The alternative is a symbol server keyed by build identifier. The debuginfod protocol is the standardised form: a client asks an HTTP endpoint for a build identifier, and the server returns the debug file, the executable, or the source file if it has them. The client is told which servers to consult through a list of base URLs, and the result is cached locally.

The protocol is deliberately simple, because it is queried from debuggers at the moment of a crash, sometimes on machines with no package manager. Each artifact type has a path derived from the identifier, and the response is the file itself. The server does the work of indexing by scanning build identifiers out of binaries as they are published, which is why a server can answer for a build it has never seen before as long as the build passed through it.

The trust model is worth stating plainly: fetching a debug file from a server means loading data that a debugger will interpret. Debug information contains DWARF expressions, which are a small bytecode; a malformed or hostile expression can cause a reader to read arbitrary memory or misbehave. That is why clients restrict which servers they consult, and why the identifier is verified after download rather than trusted from the URL alone.

Where the chain breaks

Four breaks account for most failures to resolve a crash.

The build identifier is absent. Some toolchains do not emit one by default, or the linker was invoked with the option disabled to keep output deterministic across builds. Without a note, the fallback is the debuglink name and checksum, which requires the debug file to have been shipped alongside or installed in a searched directory. A crash report from such a binary carries no identity at all, and the only recourse is to match by timestamp and hope.

The debug file was rebuilt. If the source changed and the binary was rebuilt but the archived debug file was not replaced, the identifier in the note no longer matches the debug file, and a debugger that checks will refuse it. A debugger that does not check will load it and produce line numbers from a different build — the failure is silent and worse than refusing.

The library is a different build from the one that ran. Distribution libraries are rebuilt for security updates while keeping the same file name and often the same version string, so a core file from before the update refers to addresses that no longer exist in the installed copy. The identifier in the core’s NT_FILE mapping list is the only reliable discriminator, and a debugger that resolves libraries by path alone will pick the wrong one.

The debug information is split across objects. With split DWARF, the main file’s compile units are skeletons that point at .dwo files holding the actual line tables and variable descriptions. The build identifier covers the main output, and the .dwo files have their own identifiers; an archive that has the main debug file but not the .dwo files resolves function names but not lines. The packaging must include both.

Why the identifier is worth its bytes

The alternative designs are all worse in a specific way. Naming debug files after the executable fails when two builds of the same name exist, which is the normal case on a machine that has installed several versions. Storing the debug information inside the executable defeats the purpose of stripping. Trusting a timestamp is exactly as reliable as the filesystem’s metadata and no more.

Forty hexadecimal characters — twenty bytes — is small enough to print in a crash report, paste into a URL, and use as a directory name, and large enough that accidental collisions are not a practical concern. That is the trade the format settled on, and it is why the identifier appears in core files, in package metadata, and in symbol server paths alike.

The identifier also anchors the address arithmetic that every other debugging step depends on. Line tables are only meaningful against the exact text they were built for, as described in DWARF line tables; unwind tables are only meaningful against the exact code that pushed the frames, as described in stack unwinding with call frame information. The build identifier is what makes “the exact code” a checkable claim rather than an assumption.