Write Amplification Across the Storage Stack

A write amplification factor is a ratio, and a ratio is only meaningful when both sides are specified. The database’s ratio compares bytes written to the device against bytes written by the application. The device’s ratio compares bytes written to the flash medium against bytes received from the host. The file system may have its own. When these are multiplied, the product is the number that determines how long the storage lasts and how much bandwidth a workload actually consumes, and attributing the total to the wrong layer leads to optimizations that make it worse.

Defining the numerator and the denominator

Take the denominator as the bytes the application asks to change: one row of 200 bytes, or a 4 KiB value, or a bulk load of a gigabyte. Take the numerator as the bytes the device’s controller receives. Everything between is amplification.

For a single-row update in a page-oriented engine, the chain has these terms:

• The log record for the change. A row update logs the new values and enough information to locate the old ones. For a 200-byte row, the log record is a few hundred bytes once headers and the transaction identifier are included.
• The page write at checkpoint. The page is 8 KiB regardless of how many bytes changed, so a 200-byte update contributes a full 8 KiB page write when the page is eventually flushed.
• The doublewrite copy of that page, another 8 KiB, if the engine uses one.
• The log segment write, which is the log record but written twice if the log is mirrored.
• The file system’s metadata journal, if the file grew or was allocated.
• The device’s own garbage collection and indirection-unit rewriting.

The middle two terms dominate. A 200-byte update producing 16 KiB of page writes is a factor of 80 before the device is involved, and the factor is entirely a consequence of the page being the unit of durability. The log, by contrast, is proportional to the change: a few hundred bytes for a few hundred bytes.

That asymmetry is the central fact of storage engine design. Log-structured engines minimize the page term by never rewriting a page; page-oriented engines accept it in exchange for bounded read amplification and in-place updates.

The B-tree multiplier

For a B-tree, the page term has a distribution rather than a single value. An update that finds its page already dirty in the pool costs nothing extra — the page will be written once at the next checkpoint, and the marginal cost of this update is zero. An update that dirties a page that was clean adds one page write at the next checkpoint, amortized over every update to that page before the checkpoint. And an update that triggers a split adds the writes described in page splits and write amplification.

The amplification is therefore a function of the checkpoint interval relative to the update rate on each page. A long checkpoint interval batches many updates into one page write, which lowers amplification; but it also lengthens recovery, because more log must be replayed. Engines tune this trade directly, and the tuning knob is usually expressed as a target for how much log a checkpoint may leave un-replayed.

The steady-state formula for a page-oriented engine with a full-page-write log is roughly: amplification = (page bytes per checkpoint per page) / (bytes changed per checkpoint per page) + log bytes per change / change bytes. For an 8 KiB page with a few hundred changed bytes per checkpoint, the first term is in the tens; the second is close to one. For a page with a single small change per checkpoint, the first term is in the hundreds, which is why write-heavy random-update workloads on small rows are the worst case for this architecture.

The LSM multiplier

An LSM tree replaces the page term with a compaction term. Each flush writes the memtable once; each compaction rewrites its inputs. With leveled compaction and a level ratio of ten, a byte descends through a handful of levels, so the compaction term is on the order of the level ratio times the number of levels — typically ten to thirty. That is lower than the worst case for a page-oriented engine and higher than its best case, and crucially it is predictable: it does not depend on how the writes are distributed over the key space.

The predictability is the reason LSM engines are preferred for ingestion. A page-oriented engine’s amplification depends on locality, which a random key distribution destroys; an LSM’s depends on the strategy and the level count, which are configuration. The price is the background work: compaction consumes the same device bandwidth as foreground writes, so the effective capacity is reduced by whatever fraction compaction needs. A workload that saturates the device with foreground writes will find compaction starved, the LSM will accumulate debt, and the engine will throttle.

Tiered compaction lowers the compaction term by merging only similarly sized runs, at the cost of higher read amplification and space amplification. The trade is analyzed in LSM compaction strategies, and the choice between the two strategies is a choice about which amplification the workload can absorb.

The device multiplier

Below the engine, the flash translation layer introduces its own amplification. Flash is written in pages, typically 4 KiB to 16 KiB, but erased in blocks that are much larger — hundreds of pages. An in-place update to one page requires copying the other live pages of the block elsewhere, erasing the block, and writing the survivors back. The device’s write amplification is therefore a function of the block’s occupancy at the time of garbage collection: a block that is 90 percent live costs ten times the write that triggered the collection, while a block that is 10 percent live costs close to one.

This is why the engine’s access pattern matters even when the engine’s own amplification is fixed. Writing in large sequential units lets the device pack blocks fully and garbage-collect them when they are nearly dead. Writing scattered 4 KiB pages produces blocks that are partially live for a long time, and the device rewrites them repeatedly. A file system or engine that issues 4 KiB random writes to flash can see device amplification well above what its own accounting predicts.

The mitigation is to write in units at least as large as the device’s indirection unit and to keep the write pattern sequential where possible. This is the reason log-structured designs align well with flash: their writes are already sequential and large. It is also why the doublewrite buffer’s sequential first pass, described in doublewrite buffers and torn page detection, is more valuable on flash than its durability role alone would suggest.

Attribution, and why it is hard

The three multipliers compose, but they are not independent, and attributing the total to one layer is the usual analytical error.

Consider an engine with a 10x page term writing 4 KiB random pages to a device with a 20x garbage-collection term. The product is 200x. Reducing the page term to 5x by batching updates halves the total. Reducing the device term to 10x by writing sequentially also halves it. But if the engine’s batching also makes the writes sequential — which it usually does, because batched writes go to a log or to a sorted run — then the single change improves both terms and the total drops by a factor of four. Optimizing one term in isolation and measuring the total is the only way to know which of these happened.

The measurement requires counters at each layer: bytes written by the application, bytes written by the engine (from the engine’s own statistics), bytes received by the device (from the device’s SMART or NVMe counters), and bytes written to the medium (from the device’s internal counters, if exposed). The ratio of consecutive counters is the multiplier for that layer, and the layers that are not instrumented are where the surprises are.

One term that is frequently omitted is the log’s own durability cost. A log record is written once to the log file and, at commit, forced to the device. If the log is mirrored, the write happens twice. If the log is written in small records without batching, each record costs a full device transaction, and the device’s indirection unit amplifies it further. Group commit exists precisely to amortize this, as described in write-ahead log record ordering.

Endurance as the real constraint

On flash, write amplification is not only a bandwidth cost but an endurance cost. A device rated for a finite number of program/erase cycles per block, multiplied by the block count, gives a total bytes-written budget. Divide by the amplification factor and the host’s write budget, and the result is the device’s service life under that workload. A workload writing a terabyte a day through a 20x amplification consumes twenty terabytes of the medium per day, which changes the replacement interval by a factor of twenty.

That arithmetic is why amplification is treated as a first-class design metric rather than an efficiency detail, and why engines expose it as a statistic. It is also why the choices at every layer are coupled: a page size chosen for buffer pool efficiency determines the write unit, which determines the device’s garbage-collection behaviour, which determines endurance. The page size trade itself is examined in buffer pool frames and pinning, and compression, which reduces the numerator at the top of the chain, in page compression and decoding cost.

The honest summary is that no layer can be optimized without knowing what the layer below does with the writes it receives. The read-update-memory trade-off formalized as the RUM conjecture is the reason: an access method can be tuned to improve any two of read amplification, update cost, and memory footprint, but not all three at once, and the write path is the update term. A storage engine that writes 8 KiB pages sequentially into a device that erases 4 MiB blocks has an amplification determined mostly by the device’s block occupancy; the same engine writing the same pages randomly has one determined mostly by its own page size. The engine’s design determines which of those regimes it lands in, and that is the decision the rest of this cluster has been about.