Alignment, O_DIRECT, and the Cost of Bypassing the Page Cache

Opening a file with O_DIRECT tells the kernel not to cache the data being transferred. The intent is to remove a copy and a shared cache from the path, giving the application control over what reaches the device and when. The cost is a set of constraints on every request: the buffer address, the file offset, and the transfer length must all align to the device’s logical block size, and the request must not span a boundary the device cannot handle. Requests that violate these rules fail, and library or runtime code that satisfies them invisibly is the usual reason an application appears to use direct I/O correctly without ever choosing it.

What the bypass removes

On the buffered path a read is a lookup in the page cache followed, on a miss, by a device request into a kernel page and then a copy to the application’s buffer. A write is a copy from the application into a page, a dirtying, and later a writeback by the flusher. Two copies per transfer, one on the way in and one on the way out, both sitting on top of the same physical pages the device uses.

A direct transfer skips the cache page. The application’s own pages are pinned for the duration of the transfer so that the device can address them for DMA, and the request goes to the block layer with those addresses. For a read, the only copy left is the device writing into memory; for a write, the only copy is the device reading from memory. The data’s physical addresses must be stable, which is why pinning is necessary and why page faults are not allowed during the transfer.

The pinning has a cost that is not always appreciated. Pinning a range faults in pages that are not resident, splits large pages where the hardware cannot address them for the device, and requires the page to remain unchanging until the request completes. A request built from a buffer that has never been touched pays page faults and page-fault-mandated allocations on the first pass; only reused, pre-touched buffers are actually free of that work.

The alignment rules and where they come from

The logical block size of a device is the unit it can read or write. Historically that was 512 bytes; devices with 4096-byte logical blocks are common and most devices expose a physical block size of 4096 regardless. Direct I/O requires the file offset and transfer length to be multiples of the logical block size, and the memory buffer to be aligned to it as well, because the device address must be expressed in those units.

A buffer allocated by malloc is aligned to the maximum alignment of any type in the C ABI, which is at most 16 bytes on common 64-bit targets. That satisfies a 512-byte requirement only by accident and never satisfies a 4096-byte one. The reliable instruments are posix_memalign, aligned_alloc, or mapping a page-aligned region with mmap, which is aligned to the system page size by construction. The file offset has the same constraint, which means a direct read starting at byte 1 into a file cannot be expressed at all; the request must start at a block boundary and the application must handle the unaligned head and tail through a separate buffered path.

#define _GNU_SOURCE
#include <fcntl.h>
#include <stdlib.h>
#include <unistd.h>

#define BLOCK 4096

int direct_read(int fd, off_t off, void *buf, size_t len)
{
    void *p;
    ssize_t n;

    /* Buffer, offset and length must all be multiples of the block size. */
    if (len % BLOCK || off % BLOCK)
        return -1;
    if (posix_memalign(&p, BLOCK, len))
        return -1;

    n = pread(fd, p, len, off);
    if (n > 0)
        memcpy(buf, p, (size_t)n);
    free(p);
    return (int)n;
}

The example also shows the hidden third copy: the aligned staging buffer. Borrowing an aligned buffer from the caller avoids it, which is why direct I/O APIs generally require the caller to supply one.

The remaining kernel work

Removing the page cache does not remove the filesystem. The mapping from file offset to device block still has to be consulted, extents still have to be looked up or split when the request crosses them, and extent locks still have to be taken. On a heavily fragmented file a large direct request can be split into many small device requests at extent boundaries, so the size the application asked for is not the size the device sees. A sparse region requires special handling, since reading it must return zeros rather than a physical request, and writing it must allocate blocks.

The block layer still merges and reorders, unless the file is opened with a flag that also opens the device queue directly and allows the caller to submit requests itself. Without such a bypass, the direct path is independent of cache policy but not of queue policy.

The write path also changes order. A buffered write returns as soon as the copy into the page cache is done and the data becomes durable later, at a time the application does not control. A direct write does not return until the request has completed at the device, in the sense the driver reports completion, which is not the same as durable on the medium unless the device honours a flush. For a workload that writes a large file once and never reads it, that is a reasonable trade: no cache pollution, no double copy, and the rate is the device’s rate. For a workload that expects the write to be fast because previous writes were, it is a surprise.

When it wins and when it loses

The unambiguous win is a program with its own cache that already holds the data in the shape it wants. A database with a buffer pool manages page residency itself; letting the kernel also cache the same pages doubles the memory footprint and evicts useful pages from both. The same holds for any program that reads a file larger than memory in a streaming pass and will not read it again, where caching is pure overhead that pushes everything else out.

The clear loss is small random I/O against a device with a large capacity and slow media. The page cache coalesces nothing for random access, but the readahead and write-back machinery still turn small buffered writes into efficient writeback, and the cache absorbs repeated reads of a hot region. A 512-byte random read served from cache costs roughly a copy; the same read through direct I/O is a block request and a device access.

A second loss is any program whose buffers are naturally unaligned: the staging copy puts back one of the copies the bypass removed, and the alignment logic adds work without reducing memory traffic. The pattern that helps is to allocate the buffers once, aligned, and reuse them, so the alignment cost is paid at setup.

Interaction with the page cache on the same file

The two paths are not isolated. Data written through the page cache and data written directly to the same range can be observed in either order by a reader using a different mode, and Linux’s behaviour for mixing buffered and direct I/O on the same file is the reason documentation recommends not doing it. A direct read that misses the cache can land on the device while a writeback of the same range is in flight, and the result depends on ordering at the device.

Coherence is maintained in practice by the filesystem flushing overlapping dirty pages before servicing a direct request, which is a cost proportional to how much is dirty. A program that mixes modes and finds direct I/O unexpectedly slow is usually paying that flush. Keeping one mode per file, or calling the durability primitives described in What fsync Guarantees and What It Costs on Real Devices between mode changes, avoids the ambiguity.

Direct I/O also interacts with prefetching: there is none. Each request is exactly what was asked for, so a sequential direct read loop at 4 KiB produces a 4 KiB device request per call, and the per-call overhead described in Where a read() Syscall Spends Time Before It Reaches the Data becomes the dominant term. Depth is the compensation: submitting many requests before waiting for any of them keeps the device busy, which is what the asynchronous interfaces in io_uring Submission and Completion Ring Mechanics are for.

Practical rules that survive version changes

Three rules hold across kernel versions even as the details move. Align buffers and offsets to the device’s logical block size, which is 512 or 4096 bytes, and query it rather than assuming. Reuse aligned buffers so that pinning and page faults are not part of the steady state. Keep one access mode per file, and measure with the mode the application will use, because a buffered benchmark and a direct benchmark of the same code measure different subsystems.