Topic domain
Observability
Inferring system state from the signals it emits.
- Articles
- 80
- Clusters
- 8
- Foundational
- 24
- Intermediate
- 32
- Advanced
- 24
Metrics
10Counters, histograms, and aggregation.
- Counter Resets and the Extrapolation Inside rate()
- Scrape Interval, Lookback Delta, and the Resolution of a Rate
- Histogram Buckets, the +Inf Bucket, and How Quantiles Are Interpolated
- Aggregation Laws: Which Metric Operations Commute
- Cardinality Explosion and Where It Comes From
- Exponential Histograms and the Mergeable Bucket Layout
Logging
10Structure, levels, and retention.
- Record Framing: Why a Log Line Needs a Delimiter Contract
- Structured Fields Versus Formatted Message Strings
- Severity Levels as a Filtering Contract
- Retention Policy as a Storage Budget
- Inverted Index Construction and Query Planning for Log Search
- Repeat Suppression and Template Fingerprints
Tracing
10Spans, context propagation, and sampling.
- Span Anatomy: Trace IDs, Span IDs, and Parent Links
- The traceparent Header: Field Widths and Version Negotiation
- Propagating Context Across Queues, Threads, and Proxies
- Clock Skew and the Arithmetic of Span Durations
- Head-Based Sampling: Where the Decision Belongs and What It Costs
- Tail-Based Sampling: Buffering Traces Until the Outcome Is Known
Alerting
10Symptom-based paging and burn rates.
- Error Budgets and the Burn Rate That Consumes Them
- Symptom-Based Alerting and the Cause Signals It Replaces
- Routing Alerts by Required Response: Page, Ticket, and Log Line
- Multiwindow Burn-Rate Alerts and the Short Window as a Reset
- Alert State, Hysteresis, and the Cost of Flapping
- Fleet-Wide Alerting and the Per-Instance Rule
SLO Practice
10Objectives, error budgets, and policy.
- The SLI Ratio: Numerator, Denominator, and Measurement Point
- Error Budget Units: Why Burn Rate Beats Remaining Budget
- Objective Windows: Rolling Windows, Calendar Months, and Time Equivalence
- Multiwindow Burn-Rate Alerts and Why a Short Window Also Suppresses
- Deriving Latency Targets from Percentile Distributions
- Low-Volume Objectives and the Distortion of Counting Qualifiers
Collection, enrichment, and cost control.
- Batching and Flush Policy in Telemetry Agents
- Push and Pull Collection Models and Who Owns the Sample Clock
- Head-Based Sampling by Trace-ID Threshold
- Tail Sampling and the Cost of the Decision Window
- Cardinality Control in Metric Pipelines
- Fan-Out Topologies and Duplicate Export
Hypothesis-driven investigation.
- Discriminating Tests and the Shape of a Suspect Set
- Bisecting a Deploy Range When the Fault Is Not Monotone
- Clock Domains and the Ordering of Events Across Hosts
- Reproduction Fidelity and the Sources of Nondeterminism
- Reading a Diff for the Mechanism It Changed
- Establishing Onset Before Correlating It With Changes
Continuous profiling and overhead.
- Signal-Driven Sampling: How a Profiler Captures a Stack Without Instrumenting Code
- Frame Pointers, DWARF Call Frame Information, and What Breaks When Both Are Absent
- Build IDs, Separate Debug Files, and Symbol Resolution at Scale
- Sampling Error, Skid, and How Many Samples a Profile Needs
- The Overhead Budget of a Sampling Profiler
- Folded Stacks, Call Trees, and What Aggregation Destroys
All Observability articles
- Sep 19, 2026 Inverted Index Construction and Query Planning for Log Search How tokenized log records become delta-encoded posting lists, how a time-partitioned index prunes work, and when a scan beats an index lookup.
- Sep 17, 2026 Label Matchers, Posting Lists, and How a Series Is Selected How the inverted index maps label pairs to series, why equality matchers intersect cheaply while regex matchers degrade to scans, and what the index cannot answer.
- Sep 15, 2026 Load Shedding and Retry Amplification How a collector decides what to discard when it cannot keep up, why synchronized retries sustain an outage, and how priority classes keep the useful signal alive.
- Aug 26, 2026 Objective Windows: Rolling Windows, Calendar Months, and Time Equivalence How the choice of measurement window changes what a target means, why a 28-day window is the conventional default, and how to convert a time-based target into a count-based one.
- Aug 20, 2026 Reconstructing an Incident From Partial Evidence A timeline assembled from logs, traces, metrics, and control-plane records is a model with assumptions. Separating what was observed from what the model implies is what keeps the reconstruction honest.
- Aug 10, 2026 Retention Policy as a Storage Budget How per-stream retention windows turn into bytes on disk, why compression and index size dominate the estimate, and where deletion actually costs money.
- Jul 19, 2026 Tail Sampling and the Cost of the Decision Window How a collector groups spans by trace identifier before deciding, why the wait is bounded, and what happens when spans of one trace land on different collectors.
- Jul 6, 2026 Storing Traces: Two Access Patterns That Cannot Share One Index Why a trace store must support both point lookup by trace identifier and time-range search by service, how indexes serve each, and where retention and cost force the two apart.
- Jun 30, 2026 Durable Append: Group Commit, Torn Tails, and Recovery What fsync actually orders, why a log file needs a per-record checksum, and how batching many appends into one flush changes the latency of every record.
- Jun 29, 2026 Multiwindow Burn-Rate Alerts and Why a Short Window Also Suppresses How pairing a long lookback with a short confirmation window yields both fast detection and durable alerts, and what the boolean pairing costs in edge cases.
- Jun 17, 2026 The traceparent Header: Field Widths and Version Negotiation The exact byte layout of the W3C traceparent header, why its fields are fixed width, what the flags byte means, and how a version-00 parser must treat unknown versions.
- Jun 9, 2026 Delta-of-Delta Timestamps and XOR Float Encoding on Disk How a time series block compresses to roughly two bytes per sample using a predictable timestamp pattern and the leading zeros of consecutive floats.
- May 27, 2026 Head-Based Sampling: Where the Decision Belongs and What It Costs How probabilistic and rate-limiting samplers make the keep-or-drop call at trace start, why the decision must be consistent across the trace, and how the sampled flag makes it authoritative downstream.
- May 18, 2026 Error Budget Policy: Freezes, Exemptions, and Escalation Arithmetic How a budget becomes an enforceable rule, why a freeze that everyone can override is worse than none, and how exemption quotas are computed from the budget itself.
- May 1, 2026 Durable Buffers and the Cost of At-Least-Once Export What has to be true for a queue to survive a process restart, why a write-ahead log adds a disk round trip to the export path, and what at-least-once cannot promise.
- Apr 21, 2026 The Overhead Budget of a Sampling Profiler Where the cost of continuous profiling actually goes, why per-sample cost and sample rate multiply, and how aggregation location changes the bill.
- Apr 11, 2026 Telling Saturation Apart From Contention A busy resource and a blocked one look identical in a latency graph. Queueing arithmetic, run-queue depth, off-CPU time, and steal time are what separate them.
- Mar 27, 2026 Recording Rules, Pre-Aggregation, and the Cost of Evaluating at Read Time Why the same query evaluated per-panel at render time multiplies work by viewers, how a pre-aggregated rule trades flexibility for a bounded cost, and what staleness the rule introduces.
- Mar 24, 2026 Propagating Context Across Queues, Threads, and Proxies Why an implicit current-span variable fails the moment work moves to another thread or another process, and how context has to be carried explicitly through each boundary type.
- Mar 20, 2026 Reproduction Fidelity and the Sources of Nondeterminism A failure that reproduces locally is not necessarily the failure that was observed, and the difference is a set of interleavings, clock readings, hash seeds, and externally timed events.
- Mar 19, 2026 Routing Alerts by Required Response: Page, Ticket, and Log Line How the required human response determines an alert channel, why a channel that interrupts must carry an action, and how ownership and escalation decide where a notification lands.
- Mar 17, 2026 Clock Domains and the Ordering of Events Across Hosts Why timestamps from two machines cannot be subtracted to get a latency, what NTP slewing and stepping do to a span duration, and how causal order survives without a shared clock.
- Mar 6, 2026 Histogram Buckets, the +Inf Bucket, and How Quantiles Are Interpolated Why histograms store counts per bucket rather than values, how cumulative bucket counts make merging trivial, and what linear interpolation inside a bucket assumes about the distribution.
- Mar 4, 2026 Frame Pointers, DWARF Call Frame Information, and What Breaks When Both Are Absent Two ways to recover a call stack from a captured instruction pointer, the register and table costs of each, and the failure signatures that make a stack look valid when it is not.
- Feb 7, 2026 Off-CPU Time and Scheduler Delay as a Profile How to attribute the time a thread spends not running, why the stack at the moment of blocking is the right attribution, and what makes off-CPU data heavy-tailed.
- Jan 23, 2026 Compression and the Real Cost of an Ingested Byte Why group varint framing makes a serialized batch compact, which codecs suit which signal, and how ingest pricing turns bytes in the wrong place into a bill.
- Jan 20, 2026 Error Budgets and the Burn Rate That Consumes Them How an error budget turns a service level objective into a spendable quantity, and how the burn rate compresses a window into a single number that can be compared against a threshold.
- Dec 30, 2025 Fleet-Wide Alerting and the Per-Instance Rule Why the aggregation level of an alerting rule decides both what it can detect and how many notifications it produces, and how to choose between per-instance, per-shard, and fleet-wide evaluation.
- Dec 10, 2025 Discriminating Tests and the Shape of a Suspect Set How a probe partitions the set of candidate causes, why an uninformative probe is worse than no probe, and how to price a test in bits against the time it takes.
- Nov 24, 2025 Fan-Out Topologies and Duplicate Export What happens when one agent serves several backends, why a collector behind a load balancer exports each span twice, and how to keep retries from multiplying records.
- Nov 20, 2025 Severity Levels as a Filtering Contract Why syslog orders severities from 7 downward, how numeric thresholds let a producer discard records before they are built, and what a level change does to a stream.
- Oct 31, 2025 Enrichment Order and Redaction in Flight Why the sequence of attribute processors changes the output, how a redaction rule can be undone by a later lookup, and where enrichment belongs in a pipeline.
- Sep 23, 2025 Batching and Flush Policy in Telemetry Agents How size and time triggers interact in an export buffer, what per-request overhead a batch amortizes, and why an unbounded queue is not backpressure.
- Sep 18, 2025 Merging Profiles Across a Fleet Without Losing the Tail Why summing per-host profiles answers a different question than profiling the fleet, how weight normalization changes the answer, and what a merged profile cannot tell you.
- Aug 18, 2025 Counter Resets and the Extrapolation Inside rate() Why a monotonic counter loses its meaning when a process restarts, how reset detection reconstructs the increment, and what the edge extrapolation in a rate query assumes.
- Aug 2, 2025 Confounded Comparisons Between Canary and Baseline A canary differs from its baseline in more than the code it shipped. Traffic mix, cache warmth, host hardware, and routing all shift at once, and a rare-event hypothesis needs far more requests than intuition suggests.
- Jul 28, 2025 Sampled Allocation Profiling and the Cost of Interposing on malloc How heap profilers attribute live memory to allocation sites, why sampling by size needs corrections the CPU analogue does not, and what interposition itself costs.
- Jul 23, 2025 Clock Skew and the Arithmetic of Span Durations Why span durations stay trustworthy while span start times do not, how skew makes a child appear to outlive its parent, and what correction can and cannot recover.
- Jul 17, 2025 Error Budget Units: Why Burn Rate Beats Remaining Budget How a budget derived from a target becomes a rate, why the rate is comparable across services with different targets, and where the arithmetic breaks at low volume.
- Jul 5, 2025 Cardinality Control in Metric Pipelines Why label combinations explode faster than series limits expect, how aggregation drops dimensions, and what a pipeline can do when the limit is already exceeded.
- Jul 4, 2025 Forecasting Exhaustion for Saturation Alerts How a linear projection over a recent window turns a saturation alert from a percentage threshold into a deadline, why the projection is only valid for a bounded horizon, and where it breaks.
- Jul 2, 2025 Backpressure and Loss at the Collection Boundary Why a log shipper must choose between blocking the application and dropping records, and how buffer sizing, disk queues, and drop accounting change that choice.
- Jun 3, 2025 The SLI Ratio: Numerator, Denominator, and Measurement Point Why an availability target fixes nothing until the fraction behind it is defined, and how moving the counting location changes the reported number.
- May 9, 2025 Build IDs, Separate Debug Files, and Symbol Resolution at Scale How an address in a stripped production binary becomes a function name, why the build ID is the identity that makes it possible, and where symbolication pipelines lose information.
- Apr 27, 2025 Head-Based Sampling by Trace-ID Threshold Why comparing the low bits of a trace identifier against a fraction gives an unbiased sample, and how propagation keeps the decision consistent across services.
- Feb 10, 2025 Block Compression and Columnar Layout for Log Storage Why compressing a block of records beats compressing each record, how column-wise encoding exposes value locality, and what the block size trades away.
- Jan 31, 2025 Structured Fields Versus Formatted Message Strings What is lost when a log record is a printf template, how field-value encoding preserves types, and why templates still matter for cardinality.
- Jan 1, 2025 Symptom-Based Alerting and the Cause Signals It Replaces Why paging on the metric that explains a failure produces more pages than paging on the failure itself, and how to tell a symptom from a cause when writing a rule.
- Dec 23, 2024 How Sampling Rules Decide Which Evidence Survives The telemetry left after an incident is a sample drawn by a rule chosen before the question was asked. Tail sampling, log thresholds, and retention turn the surviving data into a biased census.
- Dec 16, 2024 Alerting on Absent and Stale Data Why a missing series makes an alert rule evaluate to nothing rather than to health, how staleness markers and absence functions expose the difference, and why a firing alert can silently turn into a false resolution.
- Oct 28, 2024 Scrape Interval, Lookback Delta, and the Resolution of a Rate How sampling period sets the highest frequency a rate graph can show, why aliasing turns a steady signal into a beat pattern, and what lookback staleness does at the edges.
- Oct 15, 2024 Reading a Diff for the Mechanism It Changed A bisected revision is a pointer to a change, not a cause. Separating default-value, ordering, and resource-lifecycle differences from cosmetic ones is what turns a pointer into a mechanism.
- Oct 5, 2024 Composite Objectives: Compressing Multi-Signal Health into a Single Score How normalization, weighting, and aggregation order determine what a composite objective means, and why the same weights can hide opposite failure patterns.
- Sep 22, 2024 Folded Stacks, Call Trees, and What Aggregation Destroys Two representations of the same samples, what each preserves, and why the choice determines whether a profile can be merged across hosts or read as a flame graph.
- Sep 11, 2024 The baggage Header: Limits, Trust Boundaries, and Misuse How W3C Baggage carries arbitrary key-value pairs alongside a trace, the member and byte limits that force dropping, and why every entry crossing a trust boundary is untrusted input.
- Aug 10, 2024 Establishing Onset Before Correlating It With Changes Detection time is not onset time. Finding the minute a failure actually began, and knowing the base rate of changes in that window, is what decides whether a correlation is evidence.
- Jul 16, 2024 Bisecting a Deploy Range When the Fault Is Not Monotone Binary search over versions and inputs assumes the symptom persists once introduced; what that assumption costs, how delta debugging generalises it, and where the search stops being worth running.
- Jun 27, 2024 Deriving Latency Targets from Percentile Distributions Why a percentile objective is really a tail-count budget, how bucket layout limits the achievable precision, and what a latency target costs in headroom.
- Jun 24, 2024 Inhibition and Dependency Suppression Between Alerts How an inhibition rule mutes a set of derived alerts while an upstream alert is firing, why label equality is the whole matching mechanism, and what breaks when the suppression relation is wrong.
- Jun 11, 2024 Alert State, Hysteresis, and the Cost of Flapping How an alert moves between inactive, pending, firing, and resolved, why a condition that crosses its threshold repeatedly produces a notification storm, and what a hold duration or a second threshold does about it.
- May 30, 2024 Record Framing: Why a Log Line Needs a Delimiter Contract How a log stream is cut back into records, why newline delimiting breaks on embedded newlines, and what octet counting and escaping cost in exchange.
- May 4, 2024 Exponential Histograms and the Mergeable Bucket Layout How boundaries derived from a growth factor give uniform relative error, why an offset and a scale factor are enough to merge two histograms, and where the resolution limits sit.
- Feb 29, 2024 Cardinality Explosion and Where It Comes From How label combinations multiply into series count, why an unbounded label value destroys an index, and which containment measures actually bound the growth.
- Feb 17, 2024 Routing Traces Through a Collector Pipeline by Trace ID Why a stateful tracing stage cannot sit behind a round-robin load balancer, how consistent hashing keeps one trace on one worker, and what happens when the worker set changes.
- Jan 24, 2024 Delta and Cumulative Temporality in Collection Pipelines How the two ways of reporting a sum interact with resets, why one is cheaper to transmit and the other cheaper to align, and how a pipeline converts between them without losing increments.
- Sep 6, 2023 Counting Distinct Values Without Storing Every Record How log-frequency analysis uses register-based sketches, what the relative error formula implies for sizing, and when an exact set is the cheaper answer.
- Aug 7, 2023 Sampling Error, Skid, and How Many Samples a Profile Needs Why a profile is a binomial estimate, how instruction skid biases samples toward long-running instructions, and what confidence a given sample count actually supports.
- Jul 11, 2023 Adaptive Sampling: Feedback Control over Span Volume How a sampler adjusts its rate from observed traffic instead of a fixed configuration, why per-operation rates are needed, and the statistical damage caused by a rate that keeps changing.
- Jul 7, 2023 Aggregation Laws: Which Metric Operations Commute Why sums, counts, minimums and maximums survive regrouping while averages and percentiles do not, and how the order of rate and sum changes the answer.
- May 13, 2023 Low-Volume Objectives and the Distortion of Counting Qualifiers Why a 99.9% target on a few hundred requests a day cannot be estimated from the data, and how qualifier counting and aggregation windows change the answer.
- Apr 27, 2023 Seasonality and the Limits of Static Alert Thresholds How diurnal and weekly cycles defeat a fixed threshold, what a seasonal baseline measures against, and why differencing a series amplifies its noise in proportion to the step size.
- Apr 18, 2023 Attribution, Turnover, and the Latent Measurement Error in Objectives Why an indicator drifts away from the state it claims to measure as schemas, traffic mixes, and instrumentation change, and how to detect the drift.
- Mar 1, 2023 Multiwindow Burn-Rate Alerts and the Short Window as a Reset Why one burn-rate rule cannot be both fast and quiet, how pairing a long window with a short one separates depth from duration, and what the AND condition changes about resolution.
- Sep 5, 2022 Repeat Suppression and Template Fingerprints How collapsing identical messages into a count preserves signal, why a fingerprint must be computed over the template rather than the text, and where collapsing loses information.
- Jun 15, 2022 Tail-Based Sampling: Buffering Traces Until the Outcome Is Known Why deciding after the trace completes requires holding spans in memory, how the buffering window and the trace-ID routing rule interact, and where the pipeline loses traces it intended to keep.
- Apr 16, 2022 Signal-Driven Sampling: How a Profiler Captures a Stack Without Instrumenting Code How a timer, a signal handler, and an async-signal-safe buffer produce periodic stack samples, and why the delivery path decides what a sample can mean.
- Mar 20, 2022 Push and Pull Collection Models and Who Owns the Sample Clock Why a scrape loop and a push agent produce different failure signals, how staleness replaces absence detection, and where each model breaks down at scale.
- Feb 19, 2022 In-Kernel Aggregation: Counting Stacks in a Hash Map Instead of Shipping Them How a kernel-side profiler folds samples into a map before userspace sees them, why that removes the ring buffer from the hot path, and where the map's limits bite.
- Nov 27, 2021 Dependency Objectives: Propagating Budgets Across a Call Graph How a caller composes its own objective from the objectives of services it depends on, and why consistent fan-out makes end-to-end targets arithmetically impossible.
- Nov 22, 2021 Span Anatomy: Trace IDs, Span IDs, and Parent Links How a trace is assembled from spans, why two identifiers with different scopes are needed, and what a span record must carry for the tree to be reconstructible offline.