Retention Policy as a Storage Budget

A retention window is usually stated as a duration, thirty days or ninety days or a year, and it is actually a byte budget with a time dimension. The two are related by a rate: bytes retained equals ingestion rate multiplied by the window, adjusted for whatever reduction the storage layer applies. Getting the estimate wrong is expensive in both directions, and the errors are asymmetric: overestimating wastes money visibly, while underestimating produces a deletion event that nobody planned, on the day the volume crosses the threshold.

From a rate to a byte count

Start with the raw quantity: records per second and average record size. Both are needed, and the average is the trap. Record sizes in a log stream are not normally distributed; they are heavy-tailed. Most records are a few hundred bytes, and a minority are kilobytes because they contain a payload dump, a stack trace, or a serialized object. The arithmetic mean is dominated by the tail, and the median understates the total. The correct figure for a budget is the mean of the byte distribution, which must be measured rather than assumed from a sample of small records.

The estimate for one stream is:

retained_bytes ≈ rate_records_per_sec × mean_record_bytes × window_seconds / compression_ratio

The compression ratio is the second trap. Text logs compress well because they are repetitive: a fixed set of field names, a small set of event templates, and timestamps that share high-order digits. Ratios in the range of five to fifteen to one are typical for general-purpose compressors on log text, and the ratio depends strongly on block size, because a compressor can only exploit repetition it can see within its window. A block of a few kilobytes finds the repetition inside one record; a block of a megabyte finds the repetition across thousands of records that share a template.

There is a third term that the formula omits and that frequently dominates: the index. A system that supports full-text search stores an inverted index mapping tokens to record identifiers, and that index can approach the size of the compressed data itself. The exact ratio depends on tokenization, on whether positions are stored, and on how many distinct tokens the corpus contains. Systems that index only a small set of declared fields, and scan the rest, trade query cost for a much smaller index. Any budget that counts only the compressed payload will be wrong by a factor that depends on the indexing policy.

The shape of the curve over time

Retention is not a steady state during the first window. Bytes grow linearly with time until the oldest records reach the retention boundary; after that, ingestion and deletion balance and the total plateaus at the budget figure. The peak is the plateau, and it is reached exactly one window after the policy starts applying to a new stream.

This has two consequences that surprise operators. First, a retention change takes effect slowly: shortening a window from ninety to thirty days frees nothing for the first thirty days and then frees everything older than thirty days in a step. Second, a change to the ingestion rate propagates through the plateau over a full window, so a volume increase is visible in storage only gradually, and a volume decrease likewise.

Deletion in most storage engines is not immediate either. An append-only store marks records as expired and reclaims space during a background compaction; the space is not returned to the filesystem until compaction rewrites the affected segments. During that window, the disk holds both the old segment and its replacement, so peak usage during a large expiration can exceed the plateau. Systems that rewrite whole segments on compaction need free space equal to the largest segment being rewritten, and a policy that expires a large fraction of the data at once needs considerably more.

Tiering and the cost of a query

Retention is often implemented as a hierarchy of media rather than a single deletion date. Records move from fast storage to cheap storage to nothing, and each move changes the cost of reading them.

The trade is not primarily about bytes per gigabyte; it is about the ratio between the cost of storing a byte and the cost of reading it once. A medium optimized for sequential scan is cheap to store and expensive to query, because a query must read everything. A medium with an index is more expensive to store and cheap to query. Tiering therefore prices access patterns, not data age: a stream that is queried daily at ninety days old does not belong on the cheapest tier, and a stream that is queried only during an incident belongs there regardless of age.

A workable policy states, for each stream, both a retention window and an access expectation, and derives placement from the second. Without the access expectation, the placement rule degenerates into “move everything older than N days”, which is a policy about age that silently becomes a policy about query latency.

The second-order effect is that a query spanning a tier boundary must read from both tiers and merge. That is fine for a scan and expensive for anything requiring an ordered merge, because the slower tier sets the pace. Queries whose time range is chosen to avoid the boundary are the common mitigation, which means the boundary’s position becomes part of how the system is used.

Sampling and filtering as retention levers

Before storing less for less time, there are two cheaper reductions, and both change what questions can be answered later.

Dropping by level is the simplest: retain Error and Warning for a long window, Informational for a short one, Debug not at all. This is a statement that the value of a record depends on its severity, which is often false in a specific way: Debug records are exactly what a difficult investigation needs, and they are the records discarded first. The usual compromise is to keep a short window of everything and a long window of the severe subset, so the long window answers “was there an error” and the short window answers “what happened”.

Sampling by rate is the second lever, and its correctness depends on whether the sampled records are representative. Uniform random sampling of records preserves relative rates within a stream but destroys the ability to say that an event happened exactly once, which matters for anything involving a count of distinct identifiers or an audit trail. Sampling by template, keeping one record per template per interval, preserves the existence of rare events but loses their frequency. The two are different estimators and cannot be substituted for one another.

There is a class of record for which no sampling is acceptable regardless of volume: records that constitute a legal or security record of an action. Those have a retention requirement imposed from outside the system, and the requirement is usually a minimum, not a maximum, which inverts the usual pressure. A retention policy that treats every stream as subject to the same cost optimization will eventually delete something that had to be kept.

Deletion as an obligation

Deletion is not only a space reclamation mechanism; in many systems it is a requirement with a deadline. A record containing personal data may have to be removed on request, and a retention policy that keeps it for a fixed window is not a defence. The engineering consequence is that the storage layer must be able to find every copy of a record, including copies in backups, in replicas, and in the index.

This is where the index and the payload diverge. Deleting a record from the payload store leaves its tokens in the inverted index unless the index is also updated, and an index that stores tokens without a way to remove postings requires either a tombstone mechanism or a periodic rebuild. Tombstones are cheaper to write and require the query path to filter them; a rebuild is simpler to reason about and costs a full pass over the data.

Compaction interacts with this directly. A store that rewrites segments during compaction can drop expired records as part of the rewrite, which is efficient but means the deletion deadline is bounded below by the compaction interval. A policy that promises deletion within a day requires a compaction cadence of at most a day, and that cadence has its own cost in write amplification: rewriting a segment to remove a few records multiplies the bytes written by the fraction of the segment that is still live.

The honest summary is that a retention policy has four numbers, not one: how long records are kept, how much space that implies at the peak, how long deletion takes to become effective, and how much a query against old data costs. Stating all four makes the policy auditable; stating only the first makes it a wish.

A worked estimate

A stream at 2,000 records per second with a mean record size of 400 bytes produces about 800 kilobytes per second, or roughly 69 gigabytes per day before compression. At a compression ratio of eight to one, the payload is about 8.6 gigabytes per day. A thirty-day window is then about 260 gigabytes of payload. If the index adds fifty percent, the plateau is near 390 gigabytes, and the first thirty days of operation grow toward that figure monotonically.

The same stream at Debug level, with ten times the records, is 690 gigabytes per day raw and a plateau above three terabytes. That multiplication is the reason a level change belongs in the same review as a retention change; the two are the same budget viewed from different ends. The severity mechanism that decides what reaches storage is described in Severity Levels as a Filtering Contract, and the compression that sets the ratio in the estimate is the subject of Block Compression and Columnar Layout for Log Storage.

The estimate also needs a variance figure. A stream whose mean is 400 bytes but whose tail reaches hundreds of kilobytes has a byte total driven by a small number of records, and those records are correlated with incidents: the moment a service starts logging stack traces is the moment the volume rises. Budgeting for the mean without a headroom margin means the peak arrives exactly when the system is least able to tolerate a storage problem.