Enrichment Order and Redaction in Flight

A telemetry record produced by an instrumented process carries whatever that process chose to attach. The pipeline downstream can add more: which cluster the pod runs in, which team owns the service, which environment the deployment belongs to, whether the record contains anything that must not be stored. These operations are individually simple, and their composition is where pipelines go wrong.

Processors run in declaration order

A processor chain is a list, and the records move through it in sequence. Each stage sees the output of the previous one and nothing else. There is no dataflow analysis across stages, no reordering for efficiency, and no awareness in one stage of what another will do. The order in the configuration file is the semantics.

This makes ordering a real design decision rather than a formatting preference. Four consequences follow, each of which has produced a pipeline whose output differed from the intent.

A filter that runs before enrichment cannot filter on enriched attributes. If a database lookup adds a deployment.environment attribute and a later stage drops records from the staging environment, the filter must come after the lookup. If it comes before, it sees no such attribute and either passes everything or drops everything, depending on how it treats a missing attribute, and “missing attribute” is usually treated as “does not match”, which means passes.

A redaction that runs before enrichment can be undone. If a stage removes a field from a record and a later stage copies that field from a lookup table back onto the record, the removal is reversed. The reverse order, enrich, then redact, is the safe one, and it is the reason redaction belongs late in the chain.

Batching and enrichment interact. Enrichment is cheapest when applied once per batch rather than once per record, since the lookup result is shared by every record with the same resource attributes. A batch processor splits records into batches, and a subsequent enrichment stage operates on each batch; an enrichment before the split runs per record. The ordering rule is that expensive per-resource work belongs as late as possible, after batching has grouped records that share the resource.

Sampling discards records that later stages would have needed. A sampler early in the chain means downstream processors see a subset, and any aggregation or counting downstream counts the sample rather than the population. When a later stage’s output is used to compute a rate, the sample fraction must be applied, which requires the sampler to have recorded the probability, as described in Head-Based Sampling by Trace-ID Threshold.

Enrichment by lookup: what the join key costs

The most common enrichment is a lookup: take an identifier present on the record, such as a service name or a resource identifier from the environment, consult a table, and attach the values found. The table is either static (loaded from configuration) and refreshed on a schedule, or dynamic (queried from an inventory or orchestration API) and cached with a time-to-live.

The static case is cheap and predictable: a hash map lookup per batch, with memory proportional to the table size. Its failure mode is staleness, a service deployed after the table was last refreshed appears without its attributes, and the pipeline has no way to know whether it is missing because the table is stale or because the deployment genuinely lacks the attribute.

The dynamic case has a different cost profile. A cache miss requires a remote call, and the call is on the path of a batch. If the lookup service is slow, the batch is delayed; if the lookup service is down, the options are to pass records through unenriched or to hold them, and holding them converts the lookup outage into a pipeline backlog. A bounded cache with a negative-entry TTL is the usual compromise: unknown keys are remembered as unknown for a short period, so a burst of records for one unknown service produces one lookup rather than one per batch.

Whichever table is used, the join key must be stable across the record’s lifetime. A key derived from a pod name is invalidated by every restart, so the table grows with the number of pods ever seen and its cache hit rate falls as the workload churns. Keys derived from a workload identity, a deployment name, a namespace and service pair, are stable across restarts and produce a table proportional to the number of distinct workloads rather than the number of processes.

Redaction is a content problem, not a field problem

A redaction rule that removes a named attribute is trivially implemented and nearly useless on its own. The sensitive values that matter usually do not live in a dedicated attribute; they live inside free-form strings, a log message, a span name, a URL path, a database statement, an exception message.

The rule must therefore operate on values, not only on keys. Three shapes recur.

Structural fields with known names. Authorization headers, cookies, session identifiers, query parameters named for a token, and payment fields have predictable names across implementations. Matching on the name and replacing the value is exact and cheap. The cost is a maintained list of names, and the list is always incomplete.

Values with a recognizable format. Email addresses, credit card numbers, national identifiers, and IP addresses have patterns. A regular expression over every string attribute finds them regardless of the field name. The cost is per-byte matching on every record, which is measurable on high-volume logs, and the false positive rate is never zero, a hexadecimal trace identifier can match a pattern tuned for card numbers, so the ordering of rules matters, with the most specific format tested first.

Values with no format. A free-text field may contain a name or an address with no distinguishing syntax. No pattern finds these, and the only defensible treatment is to remove or hash the entire field rather than attempt partial removal.

Hashing is the usual substitute for deletion, because it preserves the ability to group records by the same underlying value without revealing the value. A keyed hash, a keyed digest rather than a bare one, is required for this to mean anything: a bare digest of an email address is reversible by enumerating a list of addresses and hashing each. The key must live in the pipeline’s configuration and not in the backend, and rotating it breaks joinability across the rotation, which is the cost.

import hmac, re, hashlib

EMAIL = re.compile(rb"[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Za-z]{2,}")
CARD = re.compile(rb"\b(?:\d[ -]?){13,19}\b")

def scrub(value: bytes, key: bytes) -> bytes:
    """Replace matches with a keyed digest so equal inputs stay equal."""
    def sub(m):
        digest = hmac.new(key, m.group(0), hashlib.sha256).hexdigest()[:16]
        return b"[redacted:" + digest.encode() + b"]"
    value = CARD.sub(sub, value)
    return EMAIL.sub(sub, value)

The digest is truncated to keep the replacement short; truncation to a number of hex digits that is small relative to the digest size makes collisions between distinct values possible, and the collision rate is the birthday bound over the truncated space.

The irreversible stage goes last

The property that distinguishes redaction from every other processor is that its output cannot be audited against its input. An enrichment that adds a wrong cluster name is visible the moment someone compares the record to the actual cluster. A redaction that misses a sensitive value is visible only after the value has been stored, and a redaction that removes too much is visible only as an absence.

The asymmetry argues for two placement rules. First, redaction goes as late as possible, after every stage that could reintroduce a value: enrichment, transformation, and any stage that reconstructs a field from other fields. Second, a validation stage goes after redaction, checking that the patterns no longer match, a cheap second pass over the same fields, whose failure is a counter rather than a discarded record. Without the second pass, a rule that silently stops matching because the input format changed is undetectable. The full ordering problem, of which this is one instance, is the same one described for metric labels in Cardinality Control in Metric Pipelines, where the order of relabeling and aggregation decides whether a rule can still see the label it was written against.

The same reasoning applies to the ordering of sampling and redaction. If sampling comes first, the discarded records are never redacted, and any buffer or queue that holds them before the sampling decision contains unredacted data. If redaction comes first, every record is redacted and only then sampled, which is more expensive and strictly safer. Pipelines that hold sampled-out records in a ring buffer for debugging must redact before the buffer, not after.

What enrichment costs in the bulk

Each stage in a chain reads and writes every record, and the cost per record is a few map operations plus any string matching. On a pipeline handling hundreds of thousands of records per second, a chain of eight stages is eight passes, and the passes that touch strings dominate.

The compensating factor is that most stages can be applied per batch rather than per record when the value they compute depends only on the resource. The per-resource attributes are identical across every record from one process, so a stage that computes a derived resource attribute once and attaches it to the resource is O(1) in the number of records. A stage that computes a derived span attribute runs per record. The distinction between resource-level and record-level processing is the single largest lever on a processor chain’s cost, and it is visible in the data model: attributes attached to the resource are shared, and attributes attached to the record are not.

Ordering a chain so that resource-level work happens once, and record-level work happens after batching, produces a pipeline whose cost is proportional to the number of batches for the shared work and to the number of records only for what genuinely varies. The arithmetic of how batches are formed is in Batching and Flush Policy in Telemetry Agents, and the constraint that emerges there, that per-record costs are what batching cannot amortize, is exactly the constraint that decides where an enrichment stage belongs.