Fan-Out Topologies and Duplicate Export

A pipeline rarely has one destination. A vendor backend, an internal store, an alerting path, and a compliance archive may all need the same records, and the natural way to deliver to several destinations is to fan out: one input, many exporters. Fan-out multiplies destinations without multiplying collection, which is why it is attractive. It also multiplies failure surface, and it introduces a duplication problem that is invisible until someone sums two exports and compares the total against the source.

The pipeline as a directed graph, not a chain

A collector configuration defines receivers, processors, and exporters, and connectors between them. The shape is a graph, and the legal shapes are all directed acyclic. A receiver’s output can be routed to a processor, and a processor’s output to several exporters; an exporter has no output. A connector takes the role of an exporter on one side and a receiver on the other, which is how two pipelines are composed.

Fan-out is the case of one output feeding several consumers. Its semantics are defined by what happens on delivery failure, and there are only two coherent answers.

The first is independent delivery: each exporter consumes its own copy of the batch, retries on its own schedule, and fails independently. A backend that is down stalls only its own queue; the others proceed. Memory cost is the sum of the per-exporter queues, and under a sustained failure of one backend the queue for that backend grows until it hits its limit and starts dropping. The healthy backends never see the loss, which is the desired isolation.

The second is joined delivery: the batch is considered delivered only when all consumers acknowledge it. A single unavailable backend stalls the entire batch, so the other backends stop receiving data as well. This is rarely what an operator wants, a vendor outage should not stop the internal store from receiving traces, and it is one reason it is not the default shape in most configurations.

The third option, which is really a special case of the first, is the reference-counted batch: the batch’s memory is released only when every consumer has finished with it, but each consumer proceeds independently. This reduces copying, one encoded batch is shared by all exporters rather than re-encoded per destination, at the cost of holding memory until the slowest consumer is done. Whether a given collector implementation shares or copies the encoded batch is an implementation detail that determines the memory multiplier under fan-out, and it is the reason a fan-out of four destinations does not necessarily cost four times the buffer.

Where duplicates come from

Duplicates are not a bug in any single component. They are the composite result of at-least-once delivery, retries, and fan-out, and each contributes a distinct flavour.

Retry after a lost acknowledgement. A sender transmits a batch, the receiver stores it, and the acknowledgement is lost, a dropped connection, a timeout, a proxy that closed the stream. The sender cannot distinguish this from a batch that never arrived, so it retransmits. The receiver now holds the batch twice. This is the definition of at-least-once, and a receiver that deduplicates requires an identifier it can use to recognize the second copy. The treatment of this case in the buffering layer is covered in Durable Buffers and the Cost of At-Least-Once Export.

Retry after a partial success. Some protocols report per-record status rather than per-batch status. If a batch of five hundred records has ten failures, the sender may retransmit the whole batch or only the failures, and the two choices produce very different duplicate counts. Retransmitting the whole batch is simpler and produces up to a hundred percent duplication relative to the failures; retransmitting only the failures requires the sender to track record-level state that the batch abstraction was designed to avoid.

Fan-out to destinations that overlap. Two exporters that write to the same logical store, one direct, one through a second collector that also writes there, produce two copies of every record. The topology looks like two independent paths, and it is, which is precisely the problem. Overlapping destinations are the duplication source that is hardest to find, because no single configuration file shows both paths.

Scaling out behind a load balancer. Two collectors behind a round-robin front end each receive a share of the traffic, which is fine. The problem arises when the same record reaches both: a sender that has two configured endpoints and retries against either, a load balancer with retry-on-error enabled that re-sends a request to a second backend after the first backend accepted it but replied slowly, or a DNS entry that resolves to both and is re-resolved mid-request. In each case the record is delivered twice through paths that are individually correct.

The load balancer case deserves emphasis because the fix is a configuration default rather than an application change. A proxy that retries a request after a timeout, where the original request is still being processed, duplicates every request that exceeds the timeout. The retry is only safe for requests the proxy knows are idempotent, and a telemetry export is not idempotent in the transport sense unless the receiver implements deduplication.

The trace identifier is not a deduplication key

Spans carry a trace identifier and a span identifier, and the pair is unique per span. That looks like a natural deduplication key, and using it as one is a mistake in the general case.

A span may legitimately be exported more than once by design. A head sampler at 100% and a second export path for errors produce two copies of the same span from the same process, and both are correct: they are the same span delivered to two consumers. Deduplicating on span identifier at the storage layer would silently drop one of them if the storage is shared.

A span may also be re-exported with different content. A span that is buffered, retried, and then updated, a status set after the initial record was created, produces a second record with the same identifier and different fields. Last-write-wins on the identifier is a reasonable policy, but it requires the backend to compare timestamps or versions, and a strict deduplication that keeps the first copy would preserve the stale version.

The honest statement is that identity is per-consumer. A record’s identity for deduplication is whatever the consumer defines it to be, and the exporter must not assume a global identity exists.

Bounding the duplication rather than eliminating it

Elimination is not achievable under at-least-once delivery. Bounding it is.

The most effective bound is a retry budget expressed as a ratio: retries are permitted only while the number of retried requests is below some fraction of the number of original requests over a window. Without a budget, a failing backend with many senders multiplies load exactly when the backend can least afford it, the phenomenon that makes outages self-sustaining. With a budget, the total load on the failing backend is capped at a multiple of the offered load, and the excess is dropped at the sender rather than amplified into the receiver.

The second bound is retry classification. Retrying a request that failed with a client error is pointless: a malformed payload or a rejected authentication token will fail again identically. Only a specific set of responses indicates a transient condition that a retry can cure, an explicit “too many requests” result, an unavailable-service result, a connection reset, a timeout. Retrying everything that is not a success converts a permanent failure into an infinite loop, and the loop is distributed across every sender at once.

The third bound is deduplication at the edge, where the topology makes it possible. If a single collector instance is the only path to a backend, that instance can keep a short-lived table of recently seen record identifiers, keyed by whatever the upstream sender provides and sized to the retry window. The table costs memory proportional to the arrival rate times the window, and it only works if the retry arrives at the same instance, which the load-balancing question above shows is not guaranteed.

import random, time

class TokenBucket:
    """Bounds retry traffic to a fraction of original traffic."""
    def __init__(self, rate_per_sec: float, burst: float):
        self.rate, self.burst = rate_per_sec, burst
        self.tokens = burst
        self.last = time.monotonic()

    def take(self) -> bool:
        now = time.monotonic()
        self.tokens = min(self.burst, self.tokens + (now - self.last) * self.rate)
        self.last = now
        if self.tokens < 1:
            return False
        self.tokens -= 1
        return True

def backoff_delay(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
    # Full jitter: an uncorrelated retry time avoids a synchronized second wave.
    return random.uniform(0, min(cap, base * (2 ** attempt)))

Full jitter is not a refinement; it is the difference between a retry storm and a staggered recovery. Two senders retrying on the same deterministic schedule will collide on every attempt, and the collision counts double with each transmission that fails at the same moment.

Reading duplicate rates from the pipeline itself

A duplication problem is diagnosable only if the pipeline counts the two sides. The count of records admitted at the edge and the count of records stored at the backend should agree within the sampling and filtering that the pipeline applies, and a persistent gap in either direction is the signal: stored greater than admitted means duplicates or a second path; admitted greater than stored means drops.

Neither counter is usually present by default, because both were designed as throughput counters rather than reconciliation counters. Adding them is cheap and turns the topology from an assumption into a measurement, which is the only way to confirm that a fan-out change did not accidentally create an overlapping destination.

Fan-out also interacts with sampling. A tail-sampling tier that exports to two backends halves nothing, but a policy evaluated once and a fan-out of two means the retained trace goes to both, which is usually intended. A head-sampling decision, by contrast, is made before fan-out, so a rate of ten percent at the source yields ten percent at every destination, the fan-out does not reprocess the sampling decision, and a deployment that expects independent sampling per destination must run separate pipelines rather than fan out one. The decision’s own placement at the root, and the cost of making it later, is the subject of Tail Sampling and the Cost of the Decision Window.