Write Amplification and Garbage Collection in Flash Translation Layers

A solid-state device accepts a 4 KiB write and reports it complete. Internally it may have written 4 KiB into a fresh page and later, during cleanup, moved several megabytes of live data to empty space in order to reclaim a block. The ratio between the bytes the device writes to the medium and the bytes the host asked it to write has a name, and it is the single number that predicts both how long a write burst will take and how long the device will last.

Flash can only be written in one direction

NAND cells are programmed from erased state to charged state, and a page of cells can be programmed once after the block containing it is erased. Erasing happens at block granularity, which is much larger than a page; typical page sizes are in the range of a few kilobytes and typical blocks hold a few hundred to a thousand pages. There is no way to change a written page in place. A rewrite means programming a new page elsewhere and marking the old one invalid.

That constraint produces a logical-to-physical indirection table in the device firmware, the flash translation layer. The host addresses logical blocks, typically 512 bytes or 4096 bytes; the translation layer maps each logical unit to a physical page. A write updates the mapping, and the previous physical location becomes stale. The stale pages accumulate until the block containing them is worth erasing, because a block can only be erased when every page in it is either stale or has been moved out.

The mapping granularity decides much of the cost. A page-level mapping table indexed per 4 KiB page is large: for a one-terabyte device the table holds hundreds of millions of entries, which cannot live in on-device RAM, so implementations use hybrid schemes in which a coarse table maps logical regions to physical blocks and a smaller table refines within regions. A hybrid table fragments differently from a page table, and a heavily fragmented logical address space puts pressure on the limited cache of fine-grained entries.

The garbage collection treadmill

When free blocks run low, the device chooses a victim block, copies every still-valid page in it to a free block, erases the victim, and returns it to the pool. The cost is the copy, and the amount copied is the amount of live data in the block. A block that is ninety percent stale costs almost nothing to reclaim; a block that is ninety percent live costs almost a full block’s worth of copying to free one block’s worth of space.

The amplification factor is therefore a function of how well the device’s allocation policy clusters temporally related writes. If the host rewrites a small region of the address space repeatedly, those writes land in pages spread across blocks, mixed with pages from other regions that have not changed. Reclaiming a block then requires copying the stable neighbours. The device is doing extra work because the host’s logical locality did not become physical locality.

Written by host: 8 pages of 4 KiB, all in one 4 KiB-aligned region
Device layout:   each write lands in a separate block, each block
                 also holding 31 stale-to-live pages from neighbours
Reclaim:         copy 31 live pages per block to free 1 block
Amplification:   copy volume ~ 31x the useful write volume, plus erase

The numbers in that sketch are illustrative of the mechanism, not measured values; the multiplier in any real device depends on the allocation policy, block size, and the host’s write pattern. What transfers is the direction: the more the host’s writes are scattered across the logical space relative to the device’s block size, the larger the copy volume during reclaim.

Over-provisioning is the counterweight

The device hides some capacity from the host — a few percent on consumer parts, more on enterprise parts, and more still on write-heavy parts. That hidden capacity is the slack the garbage collector uses to relocate data without stalling writes. Its effect is direct: with more spare blocks, the device can choose victims that are mostly stale, so the copy per reclaim falls. Buying a larger drive and using less of it is the same trade as buying over-provisioning, because free logical space reduces the steady-state occupancy the controller must manage.

Trimming is the interface that lets the host contribute. A modern interface command tells the device that a range of logical blocks is no longer in use, so the translation layer can invalidate the mapping without waiting for a rewrite. Without it, a deleted file’s blocks remain valid to the device and must be copied during reclaim; with it, the corresponding pages become immediately reclaimable. The cost is the command itself and the fact that a trim is a hint about future behaviour rather than an operation with a completion guarantee of media erasure.

Queue depth and the latency cliff

A flash device’s internal parallelism is high. Separate channels, separate dies, and multiple planes per die can each accept operations concurrently, and the controller schedules across them. The host’s queue depth determines how much of that parallelism is available. At depth 1, the device has at most one operation to schedule and cannot overlap internal work with the next request; at depth 32 or 64, it can.

Garbage collection changes the shape of the curve. Under sustained writes the device must interleave host requests with its own copies and erases. Erases are slow relative to reads and programs, so a burst of erases appears as a stall in host-visible latency even though the host’s queue is not empty. The result is the familiar two-level latency distribution of flash under load: most operations complete quickly and a small fraction wait for internal maintenance. Averaging over the distribution hides the stall, which is why a reporting practice that uses only mean latency misses the mechanism; the tail, and the counts of operations above a threshold, are what reveal it.

Reads are affected too, but differently. A read is a lookup in the translation table then a read of the physical page. During heavy reclaim, table misses and contention for the device’s internal bus slow reads even though no data is being rewritten for them. A read-heavy workload sharing a device with a write-heavy one therefore sees its own latency rise with the other’s write rate.

What the host can control

Three host-side decisions change the multiplier. The first is write granularity: issuing writes aligned to the device’s reported optimal I/O size and in units of at least a page avoids read-modify-write inside the device for partial pages. The second is locality: keeping the rewrite set compact in the logical address space, so that the device’s allocation policy can cluster it. Log-structured layouts do this deliberately by construction — appending to a log keeps writes sequential and stale regions large.

The third is concurrency: enough outstanding requests to let the controller schedule, but not so many that the device’s internal queues thrash. The block layer’s merging behaviour, described in Writeback Scheduling and Merging at the Block Layer, determines the sizes the device actually receives, and the ring interface in io_uring Submission and Completion Ring Mechanics determines how many are in flight.

Endurance as the same quantity

The same copy volume that costs latency costs program/erase cycles. A device rated for a number of drive writes per day is rated using an assumed amplification: if the workload’s amplification is higher than assumed, the endurance figure falls proportionally. This is why a write pattern that is benign in throughput terms can wear a device out faster than a heavier one with better locality, and why the amplification factor is worth estimating rather than assuming.

A second-order effect is that garbage collection competes with the host for the device’s DRAM. The translation table, the write buffers, and the reclaim structures all live in a small amount of on-device memory. When the working set of mappings exceeds what is cached, a host request triggers a table fetch from flash before the data can be serviced, and the latency of that fetch appears as an unexpected read cost. Device vendors size this memory for expected workloads; a workload that scatters writes across a much larger address space than expected pays for it in table misses rather than in obvious reclaim stalls.