Mapping Sampled Addresses Back to Functions and Source Lines

Every stack sample arrives as a list of numbers. On a 64-bit x86 process these are virtual addresses such as 0x55d41c2a71f3, meaningful only in combination with the memory map that was in force when the sample was taken. The step from that number to parse_header /src/server/http.c:214 is symbolisation, and it is the part of profiling that fails most quietly: a symbolisation gap does not produce an error, it produces a profile where the hot code is called [unknown].

Addresses are relative to something

A virtual address is the sum of a load bias and an offset into a file. For a position-independent executable, the kernel chooses the bias at execve time and the dynamic loader reports the resulting mapping in /proc/<pid>/maps. For a shared library, the same holds with a different bias per library. The offset that identifies code within the object file is therefore address - bias, and only that offset can be looked up in the file’s symbol table.

This is why a profile taken from one process cannot be symbolised against another process’s addresses, and why address-space layout randomisation makes raw addresses useless across runs. The profiler must record, alongside each address, enough information to identify which mapping it fell in: either the mapping’s start address and file identity, or a per-module identifier that survives restarts. Tools that store only the raw address produce profiles that symbolise correctly on the machine that produced them and nowhere else.

For kernel addresses the problem is analogous but the map is global: the kernel image is loaded at a fixed or randomised offset, and its symbol table is exported through /proc/kallsyms. Reading that file is often restricted; the kptr_restrict sysctl can cause it to report zeros for unprivileged readers, which turns every kernel frame in a profile into a blank. Restricted kernel symbols are a configuration choice, not a profiler defect, but they look identical in the output.

From offset to symbol: three tables that can disagree

Given a module-relative offset, three separate structures may answer the question “what is here”.

The ELF symbol table (.symtab) holds names and values for functions and objects. It is frequently stripped from shipped binaries, leaving only .dynsym, which contains just the symbols needed for dynamic linking. A function that is neither exported nor referenced by a relocation will be absent, so an offset may fall in a gap between two dynamic symbols and be reported as the preceding exported name, which is wrong by an arbitrary amount.

The DWARF line table (.debug_line) maps address ranges to source file and line. It covers every instruction the compiler emitted, including inlined code, so it is the only structure that can attribute a sample to a source line inside a function that has been inlined into another.

The debug information entries (.debug_info) describe subprograms and their ranges, and crucially, DW_TAG_inlined_subroutine entries describe an inlined call’s address range nested inside its caller’s range. Walking that nesting reconstructs the inline chain that the compiler flattened away.

The tables can disagree because they are generated at different stages and can be modified independently. Stripping removes .symtab but leaves .debug_* untouched, so line information still resolves while function names do not. Linker garbage collection removes code that the symbol table still mentions. A post-link optimiser can move code without updating the line table. When the two disagree, preferring the more specific answer (a line-table hit over a symbol-table range) is the usual convention, but it is a convention, not a guarantee.

Why the debug file is usually not in the binary

Debug information is large, often several times the size of the text it describes, so distributions and build systems separate it. The .gnu_debuglink section in a stripped binary names a debug file and carries a CRC32 of its contents; .note.gnu.build-id carries a hash of the original binary. A symboliser searches well-known directories for a file matching the build ID, which is a content hash, not a name, so it cannot silently match the wrong build. The build ID is the reason two rebuilds of the same source produce profiles that will not symbolise against each other’s debug files: the addresses changed even if the source did not.

# Identify a module and confirm whether it carries a build ID
readelf -n ./server | grep -A1 'Build ID'

# List the symbolised functions in a profile, restricted to one binary
perf report --symfs ./debug-root --dsos ./server --stdio

A profile that mixes builds is the classic cause of a flame graph where the same function appears twice under slightly different names, or where a function that was inlined in one build appears as a separate frame in another. The frames are not wrong; they describe two different binaries. Fixing the comparison requires keeping debug files keyed by build ID and never overwriting them, which is a storage problem more than a technical one.

Inlining, and the difference between 3% and 40%

Inlining is the single largest source of attribution error in optimised code. A small accessor inlined into a hot loop leaves no symbol of its own; the sample’s instruction pointer is inside the caller’s range, and a symbol-table-only symboliser charges the entire loop body to the caller. That is arithmetically correct at function granularity and misleading at the level people care about: the loop may be slow because of the accessor’s cache behaviour, and the profile names the loop.

DWARF fixes this by nesting. Each DW_TAG_inlined_subroutine carries DW_AT_abstract_origin pointing at the subprogram declaration, plus low and high program counters for the range where the inlining applies. A symboliser that walks the entry tree from the compilation unit down, testing each range against the sampled address, produces the full inline chain and can report the innermost frame as the accessor while still showing the call path through the loop. Without it, the flat profile is a valid function-level summary and an invalid line-level one.

The cost is that the inline expansion is not free: a deeply inlined sample requires walking several nested ranges per frame, and profiles with millions of samples and thousands of compilation units can take minutes to symbolise. This is why symbolisation is usually done once, offline, and cached by build ID.

Tail calls, PLT stubs, and frames that are not functions

Three artefacts produce names that are technically correct and practically confusing.

Calls through the procedure linkage table land in a stub whose symbol is named after the target, so a sample taken in the stub appears as a call to the target function from the target function, producing a self-referential frame. Compilers that emit -fno-plt style direct calls avoid the stub entirely, which changes the shape of the profile without changing behaviour.

Tail calls overwrite the caller’s frame. The callee returns directly to the caller’s caller, so the stack at the moment of the sample legitimately lacks the intermediate function. A profiler that assumes every call appears in the stack will report a call path that never existed in source terms. It is a correct stack and an incorrect call graph.

Jump tables and computed gotos create code regions with no corresponding source construct, and the line table may attribute them to the switch statement that generated them or to nothing. Similarly, hand-written assembly inside a C function has no line information unless the programmer supplied .loc directives.

The unknown bucket is a measurement, not a gap

A profile that reports 35% of samples as [unknown] is not 65% informative. The unknown samples are not distributed uniformly; they concentrate in exactly the code that is hardest to symbolise, which tends to be JIT-compiled code, code from stripped third-party libraries, and kernel modules without exported symbols. Reporting an aggregate percentage over the resolved subset quietly assumes the unresolved subset looks like it.

The practical response is to make the unresolved fraction visible and to break it down by module. If the unknowns cluster in one mapping, identifying that mapping (by path, by build ID, or by the fact that it is anonymous and executable) usually explains them: an anonymous executable mapping is a JIT, and its code needs the runtime’s own symbol source. That case is covered in profiling JIT-compiled code.

For samples whose stacks cannot be walked at all, the failure is upstream of symbolisation: the unwinder produced a truncated chain. That is the subject of stack unwinding, and its symptom is a stack that stops after two frames in a function known to be deep.

Checking a symbolisation before trusting the profile

A symbolised profile can be spot-checked without rerunning the workload. Resolve a known address from the profile with an independent tool and compare: addr2line -f -C -e ./server 0x1234 reports the function and line for a module-relative offset, and nm -C --defined-only lists the symbol ranges. If the two disagree with the profiler’s output, one of them is using a different bias or a different build, and the profile is suspect until that is resolved.

Two properties are worth verifying on any profile that will inform a decision. First, the fraction of samples resolved to a file and line, not merely to a function; if that fraction is low, line-level conclusions are unsupported. Second, whether any frame appears with a name that contains an address, a +0x offset, or a bare hex string. Those are the residues of a lookup that failed, and each one is a sample whose cost has been attributed to a neighbour.