mmap and the Page-Fault Path: What a Mapping Really Does

mmap does not read a file. It establishes a range of virtual addresses whose page-table entries will, when first touched, be filled in with references to pages of the file held in the kernel’s page cache. Between the call and the first access, nothing has been read from the device and no page of the file has been brought into memory. The distinction matters because it moves the cost of I/O from an explicit system call into a fault handler, where it is invisible to the program’s control flow and where the accounting and failure modes are different.

Two tables, one mapping

A process’s address space is described by a tree of virtual memory areas, each recording a start, an end, permissions, and a backing object. mmap inserts one such area, with the file and offset as the backing. The page tables are separate: they map individual pages, and they are populated on demand.

The reason for the split is that a mapping may be much larger than the memory available. A program can map a hundred-gigabyte file on a machine with sixteen gigabytes of RAM and use it correctly, because only the pages it touches occupy memory. The virtual area costs a descriptor and a bookkeeping structure; the page tables cost nothing until pages are faulted in, and even then only the touched pages.

The granularity of the whole mechanism is the page size, which is 4096 bytes on the common x86-64 and AArch64 configurations and larger on some others. The offset passed to mmap must be a multiple of it, and the kernel rounds the length up. A mapping of one byte occupies a whole page, and the bytes between the end of the file and the end of the last page read as zero.

The fault itself

The first access to an untouched page in a mapping raises a page fault. The CPU cannot resolve the address, traps, and the kernel’s fault handler runs with the faulting address. The handler looks the address up in the process’s virtual areas, finds the backing file and the offset within it, and asks the page cache for the page at that offset.

If the page is already in the cache, the fault is minor: the handler installs a page-table entry pointing at the existing page and returns. If it is not, the fault is major: the handler issues a read to the filesystem, which may go to a device, and the thread blocks for the duration. The distinction between minor and major faults is not cosmetic; it is the difference between a few hundred cycles and a disk access, and it is why the fault counts reported by /proc/<pid>/stat separate the two.

On a major fault the kernel reads a page or a cluster of pages. Linux’s readahead heuristic observes the fault pattern and increases the amount read ahead as accesses appear sequential, so a program that walks a mapping in order pays for one device read per group of pages rather than one per page. A program that walks it in a random order defeats the heuristic and pays the full cost per fault.

A write fault on a mapping that is not yet writable takes a second path. The page must exist, be marked writable in the page table, and — for a shared file mapping — be marked dirty in the page cache. A private mapping takes a third path: the page is copied, the process gets a private anonymous copy, and the file’s page remains unmodified. That copy is what makes MAP_PRIVATE a snapshot in effect for the pages that were written.

Shared, private, and the direction of the copy

The flag chooses which object the page table entry points at.

MAP_SHARED points at the page cache page itself. Stores by the process modify the cached page, and every other process mapping the same file at the same offset sees the modification, because they point at the same physical page. Writeback to the device happens later, on the kernel’s schedule or on msync, and until it completes the modification exists only in memory. A crash between the store and the writeback loses it.

MAP_PRIVATE begins by pointing at the page cache page, read-only. A read is served from the cache with no copy. The first store triggers a copy-on-write fault: the handler allocates an anonymous page, copies the contents, and installs it in the process’s page table with write permission. The file is never modified, and other processes never see the store. The cost is one page copy per page written, which for a read-mostly workload is zero and for a write-everything workload is the entire file.

The asymmetry has a practical consequence for programs that modify a mapped file and expect the change to be visible to a reader. MAP_PRIVATE writes are not visible to anyone, including after msync, which has no effect on private mappings. Only a shared mapping propagates.

#include <fcntl.h>
#include <sys/mman.h>
#include <unistd.h>

/* Map the whole file read-write, shared, and modify its first byte. */
int bump_first_byte(const char *path) {
    int fd = open(path, O_RDWR);
    if (fd < 0) return -1;

    off_t len = lseek(fd, 0, SEEK_END);
    if (len == 0) { close(fd); return -1; }

    unsigned char *p = mmap(NULL, (size_t)len, PROT_READ | PROT_WRITE,
                            MAP_SHARED, fd, 0);
    if (p == MAP_FAILED) { close(fd); return -1; }

    close(fd);          /* the mapping keeps its own reference */
    p[0] += 1;          /* store lands in the page cache page */
    msync(p, 1, MS_SYNC);  /* and is written back to the file */

    munmap(p, (size_t)len);
    return 0;
}

Closing the descriptor after mmap is correct and worth noting: the mapping holds its own reference to the open file description, so the file remains mapped until munmap or process exit. The same reference-counting rule explains why a mapping survives a fork in both processes, and why the offset within the mapping is per-address-space while the file itself is shared.

Property	MAP_SHARED	MAP_PRIVATE
Read source	Page cache	Page cache
Store destination	Page cache page	Private anonymous copy
Visible to other mappers	Yes	No
Persisted by msync	Yes	No
Fault on first store	Permission upgrade only	Copy-on-write

What msync does and does not promise

msync writes dirty pages in a shared mapping back to the file. With MS_SYNC it waits for the writeback to complete; with MS_ASYNC it schedules the writeback and returns. The call is the mapping’s equivalent of fsync, with the same limitation: it makes the data durable on the device only to the extent that the filesystem honours the request, and it says nothing about the metadata of a file whose size was changed.

The subtle point is that a store to a shared mapping marks the page dirty but does not order it against anything. A program that writes a record, sets a flag in a header page, and calls msync on the whole mapping gets durability for both, but not necessarily in the order they were written: writeback may flush the header before the record. A crash between the two flushes leaves a header claiming a record that is not there. The fix is to order the flushes explicitly, syncing the record range first and the header range second, which is the mapping analogue of the write-then-fsync-then-rename discipline.

msync is also not required for visibility. Stores to a shared mapping are visible to other processes immediately, through the shared page, without any call. The call exists for durability and for the case where a program wants to release the dirty pages so that a later munmap does not have to. Relying on munmap alone to write back is unreliable: the kernel will flush on unmap, but the timing is not something a program can observe.

Growing, shrinking, and the bus error

A mapping is fixed at creation. ftruncate on the underlying file while a mapping exists changes the set of valid offsets, and the mapping does not track the change. A page that was valid and is now beyond the new end of file becomes invalid, and the next access raises SIGBUS rather than SIGSEGV. The distinction matters for diagnosis: SIGSEGV means the address is not mapped, SIGBUS means it is mapped but the backing object no longer has data there.

The reverse direction is also a trap. Extending a file with ftruncate does not extend an existing mapping, because the mapping’s length was fixed. A program that maps a file, grows it, and continues to write past the original end writes into addresses that are not part of the mapping, or into the last partial page, which is a separate and more insidious failure: the bytes beyond the old end within the final page are legal to write but are not persisted, because they lie beyond the file’s length.

The reliable pattern is to size the file before mapping it, and to remap — munmap followed by a new mmap — when the size changes. Remapping preserves nothing; offsets and pointers into the old mapping become invalid, and any pointer arithmetic a program did across the boundary must be redone.

The address space is a resource too

Mappings consume virtual address space, and on a 32-bit process that space is scarce: a few gigabytes shared with the heap, the stack, and the executable. A program that maps many files accumulates areas, and the kernel’s limit on the number of areas per process, vm.max_map_count on Linux, defaults to 65530. A program that maps one region per record of a large dataset reaches that limit and gets ENOMEM from mmap while still having free memory.

The fix is not to raise the limit but to map fewer, larger regions. Mapping a whole file once and computing offsets into it costs one area; mapping each record separately costs one per record. The same reasoning applies to madvise calls, which are per-area operations and become expensive when there are many small areas.

MAP_NORESERVE and the overcommit policy interact here as well. A private writable mapping is charged against the commit limit as if every page would be copied, because the kernel cannot know which will be. MAP_NORESERVE waives the charge, at the cost of a possible SIGSEGV at fault time when memory is exhausted, rather than ENOMEM at map time. Neither behaviour is portable, and the default overcommit policy on Linux allows most allocations to succeed regardless.

Prefaulting and the advice interface

The lazy behaviour is usually what a program wants, but not always. A program that maps a file and then touches every page pays one fault per page, and each fault is a trap into the kernel with the associated pipeline and TLB cost. MAP_POPULATE faults the whole mapping at mmap time, turning the per-page cost into one system call; it is useful for a mapping that will be fully read once, and wasteful for one that will be partially read.

madvise gives finer control after the fact. MADV_SEQUENTIAL tells the kernel the access pattern is forward, which raises readahead. MADV_RANDOM disables readahead, which suits a lookup structure. MADV_WILLNEED starts readahead for a range without faulting it in. MADV_DONTNEED drops the pages, so a later access faults again; on a shared mapping the pages remain in the cache, and on a private mapping the dirty copies are discarded, which loses data without any error.

MADV_HUGEPAGE requests that the range be backed by large pages where the architecture supports them, which reduces TLB pressure for a large mapping at the cost of internal fragmentation. The transparent hugepage policy on Linux applies automatically to anonymous mappings above a threshold, and the advice call overrides it per range.

The interface is advisory, and the kernel may ignore any of it. A program that depends on the advice for correctness rather than performance is depending on something the interface does not promise.

When the mapping is the wrong tool

The cost model of a mapping is favourable when a file is read or written in place, repeatedly, with a working set that fits in memory, and when the program’s access pattern is close to sequential or has locality. It is unfavourable when the file is small and read once: the mmap and fault overhead exceeds a single read, and the page-cache accounting adds work for no benefit. It is also unfavourable when the program must react to I/O errors: a failed read returns an error code the caller can handle, while a failed fault is delivered as SIGBUS at an arbitrary instruction, and a handler for it cannot safely return.

The middle ground is pread and pwrite, which read at explicit offsets without disturbing the shared file position, and which leave the error handling in the caller’s control flow. The relationship between the shared position and the explicit one is part of the descriptor model described in file descriptors as table indices. A program that mixes mappings with descriptor I/O on the same file is relying on the page cache to keep them coherent, which it does, but the ordering between a store to a mapping and a subsequent read is not something the interface defines.

The fault path is also where the cost of a read on a mapped file has gone. The kernel still copies data for a plain read, from the page cache into the caller’s buffer; a mapping avoids that copy, which is the real advantage and the reason databases use it for large shared structures. The copy is a memcpy of a page, roughly proportional to the size and bounded by memory bandwidth; avoiding it matters when the same pages are read many times, and does not matter when they are read once.

The completion-based model, where a program submits an operation and collects results rather than faulting into the kernel per page, is the subject of io_uring submission and completion rings. The two are complementary: a ring can perform the read that fills the cache, and a mapping can then expose the result without another copy.