Block Compression and Columnar Layout for Log Storage

Compression ratio on log data is not a property of the compressor alone; it is a property of what the compressor is allowed to see. A record compressed in isolation can only exploit repetition within itself, and a single log record is short and internally diverse: a timestamp, a level, a message, and a handful of fields. A block of a thousand records contains the same field names a thousand times, the same template hundreds of times, and timestamps whose high-order digits are identical. The ratio differs by a large factor between the two cases, and that factor is the reason log stores are built around blocks rather than records.

The repetition structure of a log stream

Four kinds of redundancy appear at four different scales, and each needs a different mechanism.

Character-level redundancy within a record is handled by any general-purpose compressor: repeated substrings in a stack trace, repeated punctuation in a JSON object.

Field-name redundancy across records is the same key names appearing in every record. A compressor with a window of a few kilobytes sees this immediately; a compressor operating per record sees it never.

Template redundancy is the constant text of an event, which repeats with different values interpolated. This is redundancy across records separated by the variable-length values, so exploiting it requires a window large enough to span many records.

Value locality within a column is the property that all values of one field, gathered together, share a domain. A status column contains a handful of distinct integers; a host column contains a few hundred strings; a timestamp column is nearly sorted. Gathering a field’s values together exposes this redundancy, and no byte-oriented compressor can find it while the values are interleaved with other fields.

The first three are addressed by block compression with a large window. The fourth requires a columnar layout, and the two compose.

Why block size is the central parameter

A compressor’s window bounds the distance over which it can find a match. LZ77-style compressors encode a repeated substring as a back-reference with a distance and a length; the maximum distance is the window size. A window of 32 kilobytes can reference anything in the last 32 kilobytes of input, which for a 400-byte record is about 80 records. A window of several megabytes spans thousands of records and finds template-level repetition across them.

Larger blocks have three costs.

Memory. The compressor’s state grows with the window, both on the compression side and on the decompression side. A decompressor must retain the window to resolve back-references, so a reader holding several blocks open holds several windows.

Decompression granularity. A query that needs one record from a block decompresses the whole block. A 128-kilobyte block costs a decompression of 128 kilobytes to read one record; the same query against 4-kilobyte blocks costs much less. For selective queries this dominates, because the decompression is pure overhead relative to the bytes actually needed.

Latency to durability. A block is written when it is complete, so a block of a given size accumulates for a duration determined by the ingestion rate. A stream at 500 kilobytes per second fills a one-megabyte block in two seconds, which is the minimum visibility delay for records in that block. A slow stream may take minutes.

The usual resolution is a two-level scheme: blocks of moderate size for storage, with the block boundary also serving as the durability boundary, and a large-window compressor applied across a group of blocks. The group gives the long-range matching; the block gives the random-access granularity. Systems that use a shared dictionary go further: the dictionary is trained on a sample of the corpus and stored once, and every block is compressed against it, which captures cross-block redundancy without requiring the blocks to be adjacent.

The dictionary has a specific failure mode. A dictionary trained on one service’s logs compresses a different service’s logs poorly, and worse, it may expand them. A block that shares nothing with the dictionary pays for the dictionary reference without benefit. Per-stream dictionaries solve this and multiply the number of dictionaries to train, store, and version. A dictionary change is also a format change: a block compressed against dictionary version N cannot be decompressed without it, so the dictionary becomes part of the data’s durability requirement, not an implementation detail.

Columnar layout and what it exposes

Row-oriented storage keeps each record contiguous. Column-oriented storage keeps each field’s values contiguous, so a block holds all timestamps, then all levels, then all host names, and so on, each as a separate stream.

The benefit is that each column can use an encoding suited to its type and distribution.

Column kind	Encoding	Why it works
Timestamps	delta plus varint	nearly sorted, so deltas are small
Low-cardinality strings	dictionary plus integer codes	a few hundred distinct values become one- or two-byte codes
Integers in a narrow range	bit packing	values fit in a fixed number of bits, no byte padding
High-cardinality identifiers	raw or prefix-compressed	no useful distribution; avoid pretending otherwise
Text	general compressor	character-level redundancy only

Delta encoding of timestamps is the clearest win. Records arrive in time order, so the differences between consecutive timestamps are small and mostly similar, which means the deltas themselves are highly repetitive and compress further. A column of raw 64-bit timestamps costs eight bytes per record before compression; a column of deltas in the low thousands costs two bytes, and the resulting byte stream compresses well because the values repeat.

Dictionary encoding is the second win, and it has a cardinality condition. A column with C distinct values costs one code per record of ⌈log₂ C⌉ bits plus the dictionary of C values per block. For C of a few hundred, the code is under a byte, so a field that would cost ten bytes per record costs one. For C approaching the record count, the dictionary is as large as the data and the encoding adds overhead. The decision must therefore be made per column from observed cardinality, not per schema by assumption.

The third win is that a predicate can be evaluated on the compressed representation. A comparison against a dictionary-encoded column is a comparison against the dictionary, done once, followed by a scan of integer codes. A comparison against a delta-encoded sorted column can be answered by binary search over the block’s first values and a partial decode. These are the mechanisms behind the pruning described in Inverted Index Construction and Query Planning for Log Search, and they are why a columnar store can filter without materializing records.

What columnar storage costs

Assembly is the first cost. A query that returns whole records must gather one value from each column for each record, which means C reads per record for C columns, each at a different offset in a different stream. Row-oriented storage reads one contiguous run. For a query returning many records, the columnar path does more work, and the difference is proportional to the number of columns projected.

The second cost is write amplification. Records arrive one at a time, and columns are appended one value at a time, so the writer buffers a block’s worth of values per column before flushing. A block that is partially filled is either held in memory or flushed small; both are worse than the row-oriented equivalent, where a partial block is simply a shorter block. This is why columnar log stores buffer aggressively and why their visibility delay is a deliberate parameter rather than an accident.

The third cost is schema flexibility. A column exists because the schema declares it. A record with a field not in the schema must go somewhere: a catch-all column of encoded key-value pairs, which loses the per-field encoding benefit, or a separate overflow structure. Systems that accept arbitrary JSON therefore pay for it in the fields that do not fit the declared columns. This connects directly to the field-versus-schema discussion in Structured Fields Versus Formatted Message Strings: a stable field set is what makes columnar encoding effective, and an unstable one forces the general path.

Compression is not free at read time

Decompression throughput is the constraint on query speed, not disk throughput, for most log workloads. A modern general-purpose decompressor runs at hundreds of megabytes per second per core; a query scanning a terabyte of compressed data therefore spends most of its time decompressing, even at a ten-to-one ratio. This is why scan-heavy queries are parallelized by block, with one core per block, and why the achievable scan rate is roughly the decompression rate multiplied by the core count.

Two consequences follow. First, a compressor with a higher ratio but a slower decoder can make queries slower even though it stores fewer bytes; the trade is between storage cost and query cost, and the direction depends on how often the data is read. Second, the ratio achieved on a block is not a reliable predictor of query cost, because the cost is driven by the decompressed volume, which is fixed by the data, not by the compressed volume.

For archival tiers, where the data is read rarely, the ratio dominates and a slow, strong compressor is the right choice. For the hot tier, where queries are frequent, a fast compressor with a moderate ratio is usually better. Storing the same data twice at different ratios is sometimes worth it, and the retention policy described in Retention Policy as a Storage Budget is where that decision belongs.