Compression and the Real Cost of an Ingested Byte

A telemetry bill is usually quoted per gigabyte ingested, per million spans, or per series per month. Every one of those units is downstream of the same quantity: how many bytes left the pipeline and how many the store kept. Compression changes the first, and the choice of encoding changes it before compression is even applied. The two are not independent, because a compressor can only exploit repetition that the encoder preserved.

Encoding first: what protobuf does with repetition

OpenTelemetry’s wire format is Protocol Buffers. A message is a sequence of fields, each a varint-encoded key combining the field number and wire type, followed by the value. Integers are variable-length: values below 128 occupy one byte, values below 16,384 occupy two, and so on. Negative numbers stored as int32 occupy ten bytes unless the schema uses the zigzag encoding, which maps small magnitudes of either sign to small values.

Strings are length-delimited: a varint length followed by the raw UTF-8 bytes. A string that repeats across a batch, a service name, a host name, an attribute key, is written in full every time unless the schema is designed so that it is written once.

That design is exactly what the resource concept provides. A batch of spans carries one resource, whose attributes include the service name, the host, and any other attribute shared by every span in the batch. Each span then references the resource implicitly by being inside the message. In a batch of 500 spans with five resource attributes averaging 20 bytes each, the resource block occupies roughly 100 bytes once instead of 50 kilobytes across the batch.

Attribute keys repeat across records even when values do not. A span with attributes http.request.method, url.path, and http.response.status_code writes each key in full. Protobuf has no built-in string table, and a batch format that wants one must define it: a dictionary of strings at the top of the message and indices at each use site. Formats that do this compress better than protobuf with gzip, because they remove the redundancy before the compressor sees it, and the compressor’s job becomes easier at every level.

One field type deserves separate attention. The trace identifier is 16 bytes and the span identifier is 8, both effectively random. Random bytes do not compress, and they do not varint-encode usefully. They are fixed-size in the wire format for exactly that reason: any variable-length encoding would spend more bytes describing the length than the value occupies. Any pipeline that hashes identifiers into strings should be aware that hex-encoding doubles the size, and base64 raises it by a third.

What a compressor can exploit in telemetry

Compression works by finding repetition within a window. The relevant parameters are the window size, the minimum match length, and the entropy coding of literals and lengths.

Telemetry has a specific repetition profile. Attribute keys and values repeat heavily across records, the same endpoint, the same status codes, the same set of methods. Timestamps are monotonically increasing and differ by small amounts, so delta encoding makes them compress well; raw nanosecond timestamps are large numbers with high entropy in the low bytes and compress poorly without differencing. Durations are small integers with a skewed distribution and compress moderately. Identifiers are random and incompressible.

A 200-byte payload of a few records with a header, as produced by a gzip stream, frequently expands: the gzip header alone is 10 bytes, the trailer 8, and the deflate block structure adds overhead that a payload this size cannot amortize. Compression of small payloads requires a format with a minimal header or a dictionary shared out of band. This is the same arithmetic that motivates batching: the marginal benefit of compression is a function of input size, and both mechanisms attack the same constant overheads from different sides. The interaction is worked through in Batching and Flush Policy in Telemetry Agents.

Choosing a codec for the pipeline’s shape

The practical choices differ on three axes: ratio, CPU per byte, and whether a trained dictionary is supported.

Deflate, as carried by gzip and by HTTP content coding, is universally available and its ratio on a megabyte of structured text is respectable. Its cost is CPU per byte and its lack of a dictionary mechanism in the standard format. It is the safe default for compatibility.

Zstandard trades a small ratio penalty for a large throughput advantage, and its distinguishing feature is support for trained dictionaries: a sample of the workload’s own records is used to build a dictionary that is prepended to the compressor’s history. For small records, a few hundred bytes each, a dictionary changes compression from weak to substantial, because the dictionary supplies the repetition that a single record lacks. The dictionary must be generated from representative data and supplied to both sides, and a dictionary that does not match the workload degrades the ratio.

LZ4 targets speed over ratio. For a pipeline whose bottleneck is CPU rather than bandwidth, it is the right choice; for one whose bottleneck is the network or the storage bill, it leaves savings on the table.

The decision is a position on a curve, and the curve’s shape is workload-dependent. The honest statement is that the ratio differences between these codecs on telemetry payloads are large enough to matter for cost and small enough that CPU availability usually decides the choice.

The cost that compression does not reduce

Compression is applied to the payload, and the payload is not the only byte on the wire. Headers, connection establishment, TLS record overhead, and the acknowledgement traffic are all uncompressed. On a pipeline that has already batched well, these become the dominant term, and further compression of the payload yields little.

This is why the largest savings in a telemetry pipeline usually come from removing records, not from shrinking them. Sampling removes a record and everything associated with it, payload, header share, acknowledgement share, and the storage it would have occupied. An attribute dropped by a relabeling rule removes bytes from every record and shrinks the payload for every subsequent record, and it also removes a series from the store, which is where the per-series costs live, as described in Cardinality Control in Metric Pipelines.

The ordering of levers by cost-effectiveness is therefore roughly: do not produce the record; if produced, do not export it; if exported, make it smaller; if smaller, compress it; if compressed, compress more. Each step to the right costs CPU and buys less.

Where the byte count is measured

Ingest billing is computed from the bytes the backend receives, which is not the same as the bytes the sender emitted. A compressed payload is decompressed at the receiver, and the receiver may bill on either the compressed or the uncompressed size. The difference is a factor equal to the compression ratio, and it is the single most important number to establish before choosing a codec for cost reasons.

Beyond that, three measurement points in a pipeline produce three different numbers for what looks like the same traffic:

The agent’s counter is the count of bytes written to the socket, which includes the batch payload and the framing but not the transport headers unless the counter is at the transport level.

The receiver’s counter is the count after decompression, if the receiver decompresses before counting, or before it, if not. The choice is implementation-specific and generally not configurable.

The storage’s counter is the count after any filtering, sampling, and aggregation the backend applies, which may be far below the received count.

A cost model built on one of these numbers and applied to another will be wrong by the ratio between them, which can be a factor of two in either direction. The reconciliation described in Fan-Out Topologies and Duplicate Export is the same exercise applied to counts rather than bytes.

Compression as a latency source

A compressor buffers before it emits. A streaming compressor holds at least a window of input and may hold a block’s worth before producing output, so a batch handed to it arrives at the network later than the batch that was handed over uncompressed. The added delay is bounded by the block size and the codec’s flush behaviour, and it is on the order of a batch’s own transmission time for most settings.

The consequence is that the decision window of a tail sampler and the flush interval of a batch accumulator should account for compression time, and that a pipeline configured with a very large compression block on a low-traffic signal may add a delay comparable to its own batching interval. The remedy is to lower the block size or to flush explicitly at batch boundaries, both of which trade ratio for latency in the direction the operator cares about.

Decompression is cheaper than compression but not free, and it happens on the receiver, which is the component most likely to be the bottleneck at scale. A collector that decompresses every incoming payload has a per-byte CPU cost on the ingest path, and that cost is why some deployments compress only the long-haul hop between tiers rather than every hop.

The structural summary

Three statements summarize the economics. Encoding determines the redundancy available to a compressor, and a schema that shares repeated strings beats a schema that does not, before any codec is chosen. Compression ratio grows with input size, so a small batching interval wastes compression, and the two settings must be tuned together rather than separately. And the byte count that is billed is not the byte count that was produced, so any cost model must name the measurement point and be verified against the bill rather than assumed from the configuration.