Logical Block Size, Read-Modify-Write, and Misaligned I/O

The block layer’s unit of addressing is the logical block, conventionally 512 bytes and increasingly 4096. The device’s unit of programming is a NAND page, typically larger than either. Between the two sits a translation the host cannot see: the controller accepts writes at the logical granularity it advertises and assembles them into page-sized programs. When the advertised granularity is smaller than the program unit, a write that does not fill a program unit requires the controller to read the surrounding data first. The cost of that read is invisible in the interface and very visible in throughput.

Two sizes, one number

A drive’s logical block size is what the host addresses. Its physical block size, in the block layer’s terminology, is the smallest unit the device can write without an internal read-modify-write. On a modern drive the two are often 4096 bytes, in which case the device is described as 4Kn: native 4 KiB logical blocks, and no emulation. Older drives and some compatibility configurations advertise 512-byte logical blocks while using 4096-byte physical sectors, a mode described as 512e, with the e standing for emulation.

The emulation is entirely inside the device. A 512-byte write to offset 512 within a 4 KiB physical sector causes the controller to read the sector, replace the middle 512 bytes, and write the whole sector back. That is a read plus a program for what the host described as one small write. Eight consecutive 512-byte writes to the same sector cause eight read-modify-write cycles if the controller processes them one at a time, or one read and one program if it can coalesce them in its buffer. Whether it coalesces depends on whether the writes arrive close enough in time and whether the controller has buffer space to hold the sector while waiting.

The block layer exposes both numbers. The queue’s logical block size is the address unit; its physical block size is the alignment unit. A filesystem reads both and lays out its structures so that its own blocks are aligned to the physical size, which means its writes fill physical sectors exactly and no read-modify-write occurs. A filesystem that ignores the physical size and uses 512-byte blocks produces misaligned writes for every metadata update.

What misalignment costs beyond the RMW

The direct cost is the extra read, which doubles the device traffic for a partial write and consumes read bandwidth that other requests could have used. The indirect costs are larger.

First, the read-modify-write is not atomic with respect to other writes to the same physical sector. Two writes to different halves of one sector, issued concurrently, can interleave such that the second read happens before the first program, and the resulting sector contains neither write’s intended combination. The device serializes writes to the same logical block but does not necessarily serialize the read-modify-write sequences for two partial writes to the same physical sector. The block layer’s ordering guarantees are per logical block, so a host that writes two halves of a sector without a barrier between them has no guarantee about the outcome.

Second, the RMW extends the window in which a power failure leaves the sector in an intermediate state. A native 4 KiB program either completes or does not; a read-modify-write has a read, a buffer update, and a program, and a failure between the read and the program leaves the old contents, while a failure during the program leaves a partially programmed page. The device’s internal capacitors or its journaling of in-flight programs determine what is recovered, and not all devices recover the pre-write contents.

Third, discard granularity interacts. A discard request for a range smaller than the physical sector cannot be honored without affecting the rest of the sector, so devices typically reject or ignore sub-sector discards. A filesystem that frees a 512-byte fragment cannot tell the device about it, and the fragment’s physical sector stays live as far as the mapping table is concerned. The consequences of that for the device’s copy traffic are worked out in the write amplification analysis.

/* Round an extent outward to the device's physical block size. */
struct extent {
    uint64_t offset;
    uint64_t length;
};

static struct extent align_to_physical(struct extent e, uint32_t phys)
{
    uint64_t start = e.offset & ~(uint64_t)(phys - 1);
    uint64_t end   = (e.offset + e.length + phys - 1) & ~(uint64_t)(phys - 1);

    e.offset = start;
    e.length = end - start;         /* caller must handle the widened range */
    return e;
}

The widening is the catch: aligning a write outward means writing bytes the caller did not supply, which is only safe if those bytes are already correct on media or are being supplied from a read. A filesystem can align its own allocations, and it can align a direct I/O request by rejecting it, but it cannot silently widen an arbitrary application write.

Direct I/O and the alignment contract

Buffered I/O hides alignment because the page cache holds whole pages: a 512-byte write becomes a dirty 4 KiB page, and writeback issues a full-page write that is naturally aligned. Direct I/O bypasses the cache and hands the application’s buffer to the device, so the application’s offset and length become the device’s request. The interface requires the offset, the length, and the buffer address to be aligned to the device’s logical block size, and on some implementations to the physical block size or the memory page size.

The buffer address requirement is a separate constraint from the file offset. A device may use DMA that requires the buffer to be aligned, and an unaligned buffer causes either a bounce buffer copy or an error. The O_DIRECT flag’s contract on Linux is documented in the open man page: alignment restrictions apply to the offset, the length, and the address, and the specific values are filesystem- and device-dependent. Applications that need portable direct I/O query the values or align to the largest plausible size, which is the memory page size and the physical block size.

The performance argument for direct I/O is that it avoids double buffering: the data goes from the application’s buffer to the device without a copy through the page cache. On a device with a fast path and a large request, the saved copy is a meaningful fraction of the per-byte cost. The cost is that the application must handle alignment and must not expect the kernel to merge its requests with others, which loses the batching that buffered writeback provides.

Atomicity and the torn-write boundary

A write that spans a single physical sector is atomic on most devices in the sense that the sector holds either the old contents or the new, not a mixture. A write that spans multiple sectors is not: the device may program the sectors in any order, and a power failure between them leaves some old and some new. A higher layer that needs a multi-sector update to be all-or-nothing must therefore either verify the result with a checksum or arrange its writes so that each is confined to one physical unit. The size of the atomic unit is the physical sector, not the logical block, and the ordering primitives that a filesystem uses to reason about it are flush and FUA.

Some devices advertise a larger atomic unit: a guarantee that a write of up to some size, aligned, lands entirely or not at all. NVMe namespaces report an atomic write unit for normal and power-fail conditions, and the values are properties of the namespace rather than of the media. A filesystem can use a reported atomic unit to size its metadata updates so that each is atomic without a journal, which is a real optimization when the unit is large enough to hold a node. When the unit is one sector, the journal remains necessary.

The interaction with the logical block size is that the atomic unit is expressed in logical blocks. A device in 512e mode that reports an atomic unit of eight logical blocks is describing a 4 KiB physical sector, and a host that believes it has eight independent atomic units of 512 bytes has misunderstood. The guarantee is about the physical unit, and the alignment requirement is implicit.

Choosing the format

A namespace can usually be formatted with a 512-byte or a 4096-byte logical block size, and the choice is made once, often at the factory. The 4 KiB format removes the emulation and the read-modify-write path, at the cost of compatibility with software that assumes 512-byte sectors. The 512-byte format keeps compatibility and pays the emulation cost only when writes are actually misaligned, which a well-behaved filesystem avoids.

The trap is that a filesystem using 512-byte logical blocks on a 512e device can be perfectly aligned at the filesystem level and still produce misaligned device writes if its partition starts at a non-multiple of the physical sector. Partition tables have their own alignment conventions: a partition starting at sector 63, the legacy cylinder-aligned default, begins 512 bytes before a 4 KiB boundary and shifts every subsequent write by that amount. Modern partitioning tools default to a 1 MiB start for this reason, which is aligned to every plausible physical sector and erase block size.

The general rule is that alignment must be preserved through every layer: partition start, filesystem block size, allocation unit, and the device’s physical sector. A break at any one of them reintroduces the read-modify-write for the writes that follow, and the symptom is a write throughput that is a fraction of the device’s capability with no obvious cause in the workload.