Readahead Policy and the Read Sizes the Page Cache Actually Sees

Reading a file sequentially from a cold cache does not issue one device request per read call. The kernel decides, from the pattern of the last few accesses, that the reader is walking forward through the file, and begins fetching pages before they are asked for. The request size that reaches the device can be a thousand times the size of the request the program made. Measured from inside the program, though, the read is nearly instantaneous, which makes a read loop over a file a poor instrument for measuring storage. Separating the two is the first requirement for measuring either.

Detection is a state machine per open file

Each open file carries readahead state: a starting offset for the current run, a size, and a flag recording whether the last access continued the run. A read whose starting offset equals the end of the previous read’s range advances the run; anything else breaks it and starts a new one at the requested offset. The detector is deliberately cheap because it runs on every read, cached or not.

The first read in a run triggers a small fetch, on the order of a few pages, so that a program that opens a file, reads a header, and closes it does not pay for speculative data. A second sequential read doubles the window. Subsequent reads continue to grow it until it hits a ceiling set by the backing device’s maximum read size and by limits in the readahead code.

Sizes are in pages, and the growth is geometric, so the window reaches its ceiling within a handful of sequential accesses. The ceiling is where the real engineering sits: too small and the device sees many modest requests, too large and a reader that stops early has caused the kernel to fetch data nobody will look at.

The window is bounded by the device, not the file

The block layer advertises a maximum request size, max_sectors_kb in queue terms, which the readahead logic respects when it constructs its window. A file read that asks for a hundred megabytes at once does not produce a hundred-megabyte request; the kernel splits it into requests the device can handle, and the device or its driver may split further.

This is why observations from one machine do not transfer to another. A file copied on a device that accepts large requests sees fewer, larger operations and higher throughput than the same copy on a device that caps the request size lower, even though the user-space code is identical. Changes in the read pattern that appear to improve throughput are often just changes that happened to align with the device’s preferred request geometry.

The filesystem also constrains extent length. Readahead cannot fetch across a hole or a discontinuity in the extent map without issuing further lookups, so a heavily fragmented file produces more, smaller requests than its logical size implies. A file written sequentially and never fragmented maps to long runs where a single large request suffices.

What the program observes

From user space a sequential read loop sees, in the steady state, that most reads are satisfied without waiting: the page is already present because readahead fetched it. Latency per read is then a cache-hit cost plus the syscall overhead, and only occasional reads block. This is the same caching effect that makes the initial read of a file slower than all its successors.

That makes a read loop a measurement of the smallest of the following: the program’s own copy speed, the memory bandwidth of the machine, the speed of the device, and the rate at which the file’s pages can be faulted into the process address space. Which one dominates changes with the device. On a fast local solid-state device the copy loop may be the limiter; on a network filesystem the round trip dominates and the readahead window is the only thing that keeps the link busy.

A loop that reads the file twice in the same process is not measuring I/O in the second pass at all — the pages are resident — unless the file is larger than the cache or unless the pages are explicitly dropped between passes.

When readahead hurts

The failure case is any access pattern where the kernel’s prediction is wrong. Random reads produce no run, so they see the small initial fetch and no speculation; that case is benign but leaves performance entirely to the request path and the device queue depth. The damaging cases are the ones that look sequential but are not.

A program that reads a record from many files in round-robin order, a few kilobytes from each, is the classic case. Each file’s state sees a sequential-looking access, advances its window, and fetches data far beyond the region that will be read. Total device traffic rises by the ratio of the window to the useful read, while total useful bytes stay constant. The effect is worst for many small files, because the wasted prefetch is per file.

A second case is a sequential read whose destination is not local — a read that feeds a hash computation and discards the data still fetches full pages, but the fetch is not wasted since the data is used. The truly wasteful case is a reader that stops early: after reading a header the program seeks to the end of file, and the prefetched pages in between are dropped without ever being touched.

A third case is a shared device. Two processes reading different files at full window size each generate large requests that compete for the same queue, and the interleaving can turn what was sequential on the device into alternating seeks if the storage is rotational.

Controlling it

posix_fadvise provides the interface for the important cases. The POSIX_FADV_SEQUENTIAL hint doubles the readahead multiplier for the file; POSIX_FADV_RANDOM disables readahead entirely for the file, so each read fetches only what was asked for. The POSIX_FADV_WILLNEED hint starts an asynchronous fetch of a range, letting a program begin filling the cache for a region it will need soon without blocking on it. POSIX_FADV_DONTNEED releases clean pages so they can be evicted.

The manual readahead call is the direct form: it starts the fetch for a file range with no faulting or copying, so a read that follows hits the cache. The performance advantage has a limit; if the specified range is already resident, the call returns immediately, so a program can issue it unconditionally.

#define _GNU_SOURCE
#include <fcntl.h>
#include <unistd.h>

/* Ask the kernel to start fetching the next megabyte now. */
void prefetch_next(int fd, off_t offset)
{
    /* Advisory only: returns 0 even if the range cannot be fetched. */
    readahead(fd, offset, 1 << 20);
}

Direct I/O changes the arithmetic entirely, since no page cache is involved and no readahead exists: every read goes to the device with exactly the requested size. Alignment, O_DIRECT, and the Cost of Bypassing the Page Cache covers that path. Buffered writing has the mirror-image question of when the data comes back out, described in What write() Actually Does Before Data Leaves the Process.

Cache pressure, drop-behind, and competing readers

The page cache is shared. A reader with a large readahead window competes with every other page consumer, and on a machine under memory pressure the kernel cannot keep every prefetched page. Two mechanisms interact: readahead grows the window, and reclaim shrinks the cached footprint by evicting pages, including prefetched pages that have not been read yet.

The kernel counters this with drop-behind. When a sequential run is detected, the kernel marks pages before the current read position as candidates for eviction once they are consumed, so a streaming read of a file larger than memory does not push out everything else. The policy is not per-namespace or per-job: two concurrent sequential readers over the same file share one page cache, and their interleaved positions force the state machine in each direction to restart, which repeatedly shrinks both windows.

For a program that genuinely streams, the honest measurement recipe is to drop the relevant cached pages between measurement runs, or to use a file larger than the cache, or to use direct I/O, and to say which of these was done. Without one of them the reported throughput is the rate at which the kernel can copy out of memory, which is a real number but not the one the experiment usually intends. Direct I/O removes the prefetching entirely rather than tuning it, since no page cache exists for pages to be fetched into.