Remote Debugging Through gdbserver and a Target Sysroot

A debugger is not a program that observes a process from outside. It is a program that reads and writes another process’s memory and registers through the kernel’s ptrace interface, and everything it displays — stack traces, variable values, source lines — is derived from those reads plus the symbol files it has on disk. Split that arrangement across two machines and the interface changes: the debugger asks a small server on the target to perform each read and write, and the server turns each request into a ptrace call on its side.

The protocol is textual and request-response, which makes the split cheap to implement and slow to use. More importantly, it means the host debugger still needs local copies of every file whose symbols it displays. The target runs the program; the host explains it.

What runs where

gdbserver on the target is a small binary that links against the target’s C library and does three things: launch or attach to a process, service memory and register requests, and report events such as signals and thread creation. It does not read DWARF, does not know about source files, and does not need the program’s debug information. A stripped binary and a stripped libc are enough on the target side, which matters when the target has limited storage.

The host side is a full GDB built for the host architecture but able to read the target’s ELF files. A single-architecture GDB cannot do this when the architectures differ; gdb-multiarch on a distribution, or a cross-built aarch64-linux-gnu-gdb, can. The distinction is not cosmetic: the register set, the disassembler, the DWARF register mapping, and the size of a breakpoint instruction all come from the target architecture, and a host GDB configured only for x86-64 will reject a connection to an AArch64 server.

The connection is a TCP socket or a pipe, and the command is target remote <host>:<port> after loading the binary with file. When the program is started under the server rather than attached to, the server sends a stop reply immediately and waits.

How the host learns which libraries are loaded

Variable lookups need more than the main executable. A backtrace through printf needs the symbols of the target’s libc, and the addresses in the stack frames are the ones the target’s loader chose. The host therefore has to learn the target’s shared library list, including the load address of each object, before it can interpret any frame outside the main binary.

Two mechanisms supply that. The modern one is a protocol request: GDB asks the server to read a pre-formatted XML description of the loaded objects, and the server constructs it from the target’s link map. The older one, still used when the request is not supported, is a memory walk: the host reads the dynamic section of the main executable to find the DT_DEBUG entry, follows the pointer the loader wrote there into the r_debug structure, and walks the r_map linked list of link_map structures. Each node holds a name, a load address, and a pointer to the next.

Both end at the same place: a list of names and addresses. The host then opens each name in its own filesystem and reads the symbols. If the file at that path is the host’s own library, GDB will happily read the host’s symbols and describe the target’s memory using them. Frames appear, argument values appear, and they are wrong — not obviously wrong, because the structure of a libc frame is similar across architectures, but offset by whatever differences exist in the layouts.

set sysroot /opt/roots/target fixes this by prefixing every library name the target reports with the given directory. set solib-search-path adds directories to search. The sysroot must be a copy of the target’s libraries at the same versions; a root from a different distribution release gives symbols that are close and addresses that are not.

Breakpoints, and the difference between writing and watching

A software breakpoint is a write. GDB asks the server to replace the instruction at the address with a trap instruction, and the target’s ptrace performs the write. This fails in two situations that matter in cross work.

The first is read-only text. Code in flash, in a read-only mapped segment, or under an MMU configuration that forbids writes returns an error, and GDB reports that it cannot insert the breakpoint. Hardware breakpoints use the core’s debug registers instead, are limited to a small number — four is a common count on embedded cores — and work regardless of page permissions. hbreak selects them.

The second is an address that is not mapped yet. A breakpoint on a function in a shared library set before that library is loaded points into unmapped memory, and the insert fails or the trap is never reached. Deferred breakpoints solve this: GDB records the request by name, sets it when a library containing the symbol is reported as loaded, and reports the pending state until then. The setting that controls whether an unresolvable name is an error or a pending request is set breakpoint pending on.

Position-independent executables complicate the same issue from the other side. The addresses in the symbol table are relative to a load base chosen at run time, and GDB relocates them once it knows the base that the server reports. A breakpoint set before the connection knows the base is placed at the link-time address, which is usually unmapped. Setting breakpoints after the connection is established and the object list has arrived avoids the problem; breaking at main by name rather than by address does so implicitly.

Address space layout randomisation belongs to the target. Whether a given launch disables it depends on the server and on how the process was started, and a process attached to after it began keeps whatever layout the kernel gave it. Debugging a crash that depends on a specific address therefore requires controlling the launch, not the debugger.

Signals, threads, and the all-stop model

The remote protocol reports every thread separately, and info threads on the host shows the target’s thread list with the server’s identifiers. In the default all-stop mode, any event stops every thread in the process, the host is told which thread caused it, and resumption applies to all of them. Non-stop operation exists in the protocol and is not uniformly supported by servers on small targets, which is why remote sessions on embedded systems usually behave in the older, simpler way.

Signals cross the boundary as events. A SIGSEGV that the program handles itself will still stop the session unless the host is told to pass it through: handle SIGSEGV nostop noprint pass. Debugging a program that installs handlers for SIGUSR1 or SIGCHLD without that configuration produces stops at points that have nothing to do with the bug being investigated, because the debugger intercepts every delivery before the handler sees it.

The debugger’s own view of time is the target’s, and single-stepping a remote process is one round trip per instruction. Stepping through a loop of any length is impractical, and the usual approach is a breakpoint after the loop or a conditional breakpoint evaluated on the target. Whether conditions are evaluated on the target or by the host depends on the server’s support for conditional breakpoint packets; when it is absent, every hit stops the process and the host evaluates the condition, which turns a conditional breakpoint into a sequence of stops.

Symbols from a build that no longer exists

The most common reason a remote session shows the wrong source line is a mismatch between the binary running on the target and the file loaded on the host. Build identifiers catch this when the toolchain records them: GDB compares the note in the running image with the one in the local file and warns when they differ. A build with -Wl,--build-id produces the note; the corresponding debug file is found under /usr/lib/debug/.build-id/ by its hash, or beside the binary through the .gnu_debuglink section.

set debug-file-directory points at a directory tree holding separate debug files, which is the arrangement distributions use to ship stripped libraries and a parallel tree of symbol files. For a cross build, the target root provides the stripped libraries and a matching tree provides the symbols; both must come from the same build of the same package. Mixing a symbol file from a rebuild with a binary from an earlier one produces plausible source lines and wrong variable values, which is worse than no symbols at all.

Core dumps follow the same rule. A dump taken on the target can be analysed on the host with the binary and the sysroot, and it requires exactly the same three inputs: the executable with symbols, the sysroot for libraries, and a GDB that understands the target’s architecture. The advantage is that the dump is a static file, so the analysis does not depend on the target staying up or on the network.

Where the boundary is a hard limit

Some debugging tasks cannot cross the boundary. Kernel debugging needs a different interface entirely, since the debugger must control the kernel rather than a user process. Driver code whose failure is an interrupt storm leaves the server unable to answer requests. A program that crashes the kernel takes the server with it, and the last thing the host sees is a dead connection with no stop reply.

For those cases the alternatives are a full-system emulator with its own debugger interface, a hardware debug probe attached to the target’s JTAG or SWD pins, or instrumentation in the program. The remote protocol is a good fit for user-space logic and a poor fit for anything below the system call boundary. What it does well — reading a stack, inspecting a structure, setting a breakpoint in a function that has not been called yet — it does with a small server and a copy of the target’s files, and that combination is the same one a cross build already has on disk: the sysroot described in what a sysroot actually restricts and the toolchain that produced the binary.