Conditional Breakpoints: Where the Test Runs and What It Costs

A breakpoint with a condition attached looks like a filter. It stops only when the expression is true, so a breakpoint in a loop that runs a million times and matches once appears to cost one stop. That appearance is wrong. The trap instruction still fires on every iteration; the debugger still regains control, evaluates the expression, decides the condition is false, and resumes. The condition reduces the number of visible stops, not the number of stops, and the difference determines whether the technique is usable in a hot path.

The stop is the unit of cost

Every hit of a conditional breakpoint costs a full stop-and-resume cycle: the trap, the kernel’s transition to the tracer, the tracer’s evaluation of the condition, and the resume. On a local process that is two context switches plus the evaluation. Over a remote connection it is also a network round trip, and over a serial link at 115200 baud the raw channel moves roughly eleven kilobytes per second, so each packet exchange costs on the order of a millisecond before any work is done.

The consequence is a simple inequality. The cost of a conditional breakpoint is the number of times the address is executed multiplied by the cost of a stop, plus the cost of evaluating the expression on each of those. It is not the number of times the condition is true multiplied by anything. A condition that is true once in ten million iterations costs ten million stops.

Two features exist to make this bearable, and both are often confused with filtering. An ignore count, set with ignore 3 5000, skips the first five thousand hits by decrementing a counter in the debugger. It still stops five thousand times; it only suppresses the report. A command list with silent and continue likewise stops on every hit and resumes automatically, which is enough to make the output readable but not to make the breakpoint cheap.

What the condition does buy is precision. The expression is evaluated in the context of the stopped frame, so it can read locals, dereference pointers, and call functions. A watchpoint can observe a change but cannot ask a question about the surrounding state; a conditional breakpoint can, at the price of stopping at every execution of the address rather than at every write.

Evaluating an expression without a compiler

The condition is written in the debugger’s expression language and has to be evaluated in the target’s context, where the variables live. There are three places the evaluation can happen, and the choice is usually invisible to the person who typed the condition.

The first is the debugger itself. The tracer reads the variable’s value from the tracee’s memory or registers and computes the result locally. This requires one or more memory reads per evaluation, each of which is a ptrace call locally or a protocol round trip remotely. A condition touching three variables may be three reads plus the expression tree walk.

The second is the target, through compiled bytecode. GDB translates expressions that consist of arithmetic, comparisons, and memory dereferences into a small stack-machine program, and a cooperating stub can execute it without returning control to the debugger. The stub reads its own memory, which is cheap, and answers only when the condition is true or when it needs something it cannot compute. This removes the round trip per evaluation but not the trap per execution; the stop still happens, it is just shorter.

The third is the program itself, which is the only option that removes the trap, and it is the subject of the last section on moving the test into the code.

Whichever path is used, the expression must be valid at the address where the breakpoint sits. That is a stronger requirement than it appears. A local variable may be in a register at one point in the function and on the stack at another; the debug information describes which, per address range. A condition evaluated at an address where the variable has no location cannot be answered, and the debugger reports the variable as unavailable rather than guessing. This is one of the ways optimised code makes conditional breakpoints unreliable: the condition may be evaluable at the function entry and not inside the loop.

Conditions that call functions

An expression may contain a function call: a comparison against the result of a getter, a check on a container’s size method, a predicate. Evaluating it requires running code in the inferior, which is a different operation from reading memory.

The debugger saves the thread’s registers, writes a call frame onto the target’s stack, sets a breakpoint at a return address it controls, sets the program counter to the function, and resumes. On x86-64 the frame must be placed below the 128-byte red zone that a leaf function is allowed to use without adjusting the stack pointer, or the call will overwrite the data of the function it was called from. The call runs as part of the program, with the program’s locks held or not held, its global state, and its side effects. When it returns, the debugger reads the result and restores the context.

Three failure modes follow directly. The call may have side effects the condition’s author did not intend, which is why debuggers warn about calling functions in conditions. The call may block on a lock held by another thread that the debugger has stopped, producing a deadlock that no timeout resolves. And the call may not return at all: a function that exits, throws, or calls longjmp leaves the debugger’s synthetic frame in an inconsistent state.

On a remote target the same operation is worse, because the debugger must write the call frame through the protocol and then run the target until the return breakpoint fires. The cost is no longer per instruction but per call, and a condition evaluated on every hit of a hot breakpoint with a function call inside it is a reliable way to make a program appear to hang.

# The condition is evaluated at every hit; the print happens only when true.
break process_item if item->state == STATE_DIRTY
commands
  silent
  printf "dirty item %d\n", item->id
  continue
end

Moving the test into the program

The alternative that actually removes the cost is to let the compiler emit the test. A guard in the source, with a breakpoint inside the guarded block, becomes an ordinary conditional branch in the instruction stream:

if (item->state == STATE_DIRTY) {
    inspect(item);   /* breakpoint here, reached only when dirty */
}

The branch is predicted, costs a few cycles when not taken, and the breakpoint fires only when the condition holds. The trap count drops from every iteration to every match. The cost moves to the program: the guard is now part of the code being tested, it must be removed before shipping, and it can change optimisation decisions in its neighbourhood. It also cannot be added to a binary that is already running, which is the case where a debugger earns its keep.

Between the two extremes sit the kernel’s dynamic probes. A probe attached to a function or an address can carry a filter that is evaluated in kernel context, before any notification reaches a user-space tool, so a match that occurs once in a million calls produces one event rather than a million. The filter is compiled, not interpreted by a debugger, and the mechanism is available only where the kernel provides it and only for the kinds of expressions it supports.

For anything that must observe a value changing rather than an address executing, the processor’s watchpoints are the alternative, and they have the opposite cost profile: no trap unless the watched range is touched, but only a handful of ranges can be watched at once, as described in hardware breakpoints and watchpoints. Choosing between the two is a question of which event is rarer: executions of the address or writes to the variable.

What the condition cannot observe

The condition is evaluated after the stop, so the state it inspects is the state at the breakpoint, not at the moment the variable changed. If the breakpoint sits after the assignment that made the condition true, the intervening code has already run. Placing the breakpoint earlier changes which iteration is caught, and on optimised code the earlier address may not exist.

Conditions are also evaluated on the thread that hit the breakpoint, one thread at a time. A condition that is true only in the presence of a concurrent write will not be observed unless the other thread’s write has already landed and is visible, which depends on memory ordering and on how much of the program the debugger has stopped. The breakpoint does not synchronise anything; it only serialises the thread that hit it.