Observability · Cluster
Profiling in Production
Continuous profiling and overhead.
- 01 Signal-Driven Sampling: How a Profiler Captures a Stack Without Instrumenting Code How a timer, a signal handler, and an async-signal-safe buffer produce periodic stack samples, and why the delivery path decides what a sample can mean.
- 02 Frame Pointers, DWARF Call Frame Information, and What Breaks When Both Are Absent Two ways to recover a call stack from a captured instruction pointer, the register and table costs of each, and the failure signatures that make a stack look valid when it is not.
- 03 Build IDs, Separate Debug Files, and Symbol Resolution at Scale How an address in a stripped production binary becomes a function name, why the build ID is the identity that makes it possible, and where symbolication pipelines lose information.
- 04 Sampling Error, Skid, and How Many Samples a Profile Needs Why a profile is a binomial estimate, how instruction skid biases samples toward long-running instructions, and what confidence a given sample count actually supports.
- 05 The Overhead Budget of a Sampling Profiler Where the cost of continuous profiling actually goes, why per-sample cost and sample rate multiply, and how aggregation location changes the bill.
- 06 Folded Stacks, Call Trees, and What Aggregation Destroys Two representations of the same samples, what each preserves, and why the choice determines whether a profile can be merged across hosts or read as a flame graph.
- 07 Merging Profiles Across a Fleet Without Losing the Tail Why summing per-host profiles answers a different question than profiling the fleet, how weight normalization changes the answer, and what a merged profile cannot tell you.
- 08 In-Kernel Aggregation: Counting Stacks in a Hash Map Instead of Shipping Them How a kernel-side profiler folds samples into a map before userspace sees them, why that removes the ring buffer from the hot path, and where the map's limits bite.
- 09 Off-CPU Time and Scheduler Delay as a Profile How to attribute the time a thread spends not running, why the stack at the moment of blocking is the right attribution, and what makes off-CPU data heavy-tailed.
- 10 Sampled Allocation Profiling and the Cost of Interposing on malloc How heap profilers attribute live memory to allocation sites, why sampling by size needs corrections the CPU analogue does not, and what interposition itself costs.