Build IDs, Separate Debug Files, and Symbol Resolution at Scale

A profile records addresses. Turning those into function names, file names, and line numbers is a lookup that seems trivial until the first production deployment: the binary running on the host has been stripped to save space and to avoid shipping source paths, the debug information sits in a separate file that may not be on the same machine, and the address the profiler recorded is not the address the linker wrote into the symbol table.

Each of those three facts has a mechanism behind it, and each produces a specific class of wrong answer when ignored.

Three address spaces that all look alike

The compiler emits code at addresses relative to the start of the section it is placing. The linker combines sections from many object files and lays them out at a virtual address chosen by the program headers, so the address in the final executable is the section’s virtual address plus the offset within it. At runtime, a position-independent executable is loaded at whatever base address the kernel’s loader chooses, and the recorded instruction pointer is the load base plus the link-time address.

A profiler that records raw instruction pointers therefore records addresses that differ between two runs of the same binary on the same machine, because address space layout randomization moves the base. The correction is to subtract the load base of the mapping containing the address, recovering a link-time address that is stable for that build.

The base is discoverable from the process itself. /proc/<pid>/maps lists each mapping with its start address, its offset in the underlying file, and its path. For a PIE executable the relevant mapping is the one whose path matches the binary and whose offset is zero; its start address is the base, but only if the mapping corresponds to file offset zero. A segment mapped at a nonzero file offset has a base that must be computed by subtracting that offset. Getting this wrong shifts every symbol by the size of the preceding segment, which produces profiles where the top functions are plausible neighbours of the real ones.

Shared libraries each have their own base and their own identity. A profile of a process that spends its time in a system library needs to resolve addresses in that library’s address space, using that library’s file and that file’s build identity. Any design that assumes one binary per process breaks on the first program that links against a dynamically loaded library.

The build ID is the only durable identity

A path is not an identity. The same path on the same host may hold different builds before and after a deploy, and a symbol server that keys on path alone will hand back the wrong debug information for older samples. A version string is not an identity either, because it is assigned by a human and does not change when the compiler’s output changes.

The build ID is. On ELF systems, the linker can embed a note describing a unique identifier for the link output, and GNU ld and gold do this by default on most distributions. The note lives in a section named .note.gnu.build-id, and its content is a hash of the linked output. The kernel records the same identifier in /sys/kernel/notes for itself, and the dynamic loader does not care about it, which is why stripping tools preserve it.

Two consequences follow. First, the identifier travels with the binary, so a debug file can be keyed on it and a lookup becomes exact. Second, the identifier is computable from the binary at the time it is built, so a pipeline can publish debug information before the binary is deployed, and the profiler can resolve an address the moment it sees one.

A build ID is not a security mechanism. It is a hash with no signature, and a symbol server that trusts an uploaded identifier without checking the accompanying file is trusting whatever the uploader claimed. The identifier’s job is disambiguation, not authentication.

Splitting debug information from the shipped binary

Debug information is large. Line tables, variable descriptions, and type information commonly exceed the size of the code they describe, sometimes by a wide margin, and none of it is needed to execute. The standard arrangement is to ship the stripped binary and keep the debug information in a separate file.

objcopy --only-keep-debug produces a file containing the sections needed for debugging and nothing else. objcopy --strip-debug removes those sections from the binary. objcopy --add-gnu-debuglink appends a small section to the stripped binary recording the name of the debug file and a checksum of its contents. Debuggers and symbolizers use that section to find the file when it sits beside the binary, and fall back to a search path or a network service when it does not.

The split is a trade with three axes. Space: the shipped image shrinks by the size of the debug sections, which matters when the artifact is copied to every host in a fleet. Information: the stripped binary still contains the symbol table unless --strip-all is used, so function names may survive even without debug information, while line numbers and inlined call sites do not. Recoverability: the debug file must be retained, keyed by build ID, for as long as any sample referring to that build can appear in storage.

The third axis is the one that decays. Retention policies written for log data, measured in days, are far too short for debug artifacts, because a profile store holding samples from the previous quarter will still contain addresses from builds that have been replaced three times. The usual failure is not a missing pipeline but an expired artifact: symbolication works during the deploy window and silently degrades to raw addresses later, and the degradation is invisible unless the pipeline counts unresolved samples.

What a symbolizer does with an address

Resolution proceeds in stages, and each stage has a different requirement on the available information.

The first stage is file and section identification: given an address and a base, find the object and compute the link-time address. This needs only the mapping table, so it works even when no debug information exists.

The second is symbol lookup. The .symtab section, when present, maps address ranges to names; .dynsym holds the subset needed for dynamic linking and is present even in fully stripped shared objects, which is why exported functions often resolve while internal ones do not. Both are sorted arrays of fixed-size records with a string table, so lookup is a binary search over a few tens of thousands of entries, and the cost is dominated by the string table access rather than the search.

The third is line lookup, which requires the DWARF line program. The line program is a bytecode that describes how to build a matrix of (address, file, line, column) rows; the profiler runs it once per compilation unit and then binary-searches the resulting rows. This is where inlining changes the answer: a single address may belong to several nested logical functions, and DWARF records the inline chain as a sequence of DW_TAG_inlined_subroutine entries with call-site locations. A symbolizer that reports only the innermost function discards the inline structure, and a profiler that reports only the outermost one attributes work to a function that never executed as a frame. Both choices are common, and the difference between them can be most of the profile for code with heavy inlining.

addr2line -f -C -i -e /path/to/binary.debug 0x4011a6

The -i flag is the one that expands inline frames; without it the tool reports the outermost function containing the address. The -C flag demangles C++ names. Neither flag changes the address computation, which is why an address that resolves to a plausible name can still be off by a whole function if the base was computed wrong.

Caching, and the cost of asking twice

Symbolication is expensive enough to be done once per unique address rather than once per sample. The natural cache key is the tuple of build identifier and link-time address, and that key is stable across hosts, across restarts, and across time. A cache keyed on process identifier or on raw instruction pointer is worthless, because both change on every run.

The volume is the reason caching is mandatory rather than an optimization. A profile with a hundred thousand samples and an average depth of thirty frames produces three million address references per collection interval, and resolving each one against a DWARF line program would dominate every other cost in the pipeline. Deduplicating to unique (build, address) pairs typically reduces the count by orders of magnitude, because a program spends its samples in a small fraction of its code.

There is a subtler cost in the resolution itself: the debug file must be fetched, decompressed, and its line programs parsed. Fetching per request is untenable, so implementations keep a decompressed index or an on-disk cache of resolved ranges. The trade is between memory and fetch latency, and the failure mode of the memory side is a symbolizer that grows without bound as it accumulates builds, which is the same retention problem in a different place.

What remains unresolved

Some addresses cannot be resolved, and the reasons are worth distinguishing because they call for different fixes. An address in a region with no file mapping belongs to anonymous executable memory, which is JIT output or generated trampolines; it needs the runtime’s own symbol source, not the ELF pipeline. An address whose build identifier is unknown means the artifact was never published, which is a pipeline gap. An address whose build identifier is known but whose debug file has expired is a retention gap. An address that resolves to a function but not to a line means the debug information was compiled without line tables, which is a build flag.

A profile that reports the fraction of samples in each of those categories is diagnosable. A profile that reports only names, with unresolved frames silently rendered as hexadecimal, hides all four behind the same symptom: a suspiciously large amount of time in a function called [unknown].

Once addresses resolve to names, the profile is still a set of independent samples. The next problem is how much confidence a given number of samples supports, which is treated in Sampling Error, Skid, and How Many Samples a Profile Needs.