Memory-Mapped Persistence: What msync and DAX Actually Guarantee

A persistent memory file is opened, mapped with MAP_SHARED, and the program writes to it with ordinary stores. The writes are visible to every process mapping the same file, which is stronger than the usual page-cache story, and the data reaches the DIMM without a system call. The convenience hides a boundary: the mapping makes bytes durable, and the filesystem makes the file’s size, its block allocation, and its name durable. A crash between those two makes leaves bytes on the media that belong to no file, and a crash in the other order leaves a file whose length promises data that was never written.

A shared mapping is a window, not a copy

mmap with MAP_SHARED installs the file’s page-cache pages into the process’s page tables. A store to the mapping modifies the page cache directly; there is no copy to flush later in the sense of a private mapping, and every other process that maps the same page sees the change immediately through the same physical page. MAP_PRIVATE is the different case: the pages are copy-on-write, a store triggers a fault and an anonymous copy, and the modification is never written back to the file at all.

That immediacy is the source of the confusion. A store through a shared mapping has modified the file’s cached contents, which is a stronger statement than “the write was issued”, but it is not a statement about media. The page is dirty in the page cache, and the writeback machinery will schedule it like any other dirty page. A crash before writeback loses it. The mapping changes where the store lands, not whether it is durable.

msync is the call that asks for writeback. With MS_SYNC, it writes the modified pages in the range and waits for them to reach the device. With MS_ASYNC, it schedules the writeback and returns. MS_INVALIDATE asks the kernel to invalidate other mappings of the same pages so that subsequent reads refetch, which matters when the file is also being modified by another path. The range is rounded to page boundaries, and a range that does not cover every modified page leaves the rest dirty, so a program that calls msync on a subrange and assumes the whole file is durable is wrong in the same way as a program that calls fsync and assumes the directory entry is durable.

msync is not fsync, and the difference is metadata

fsync on a file descriptor flushes the file’s data and the inode metadata that describes it, including the size. msync on a mapping flushes the pages that the mapping covers, which is data and whatever the file’s block mapping already records, and it does not flush the inode. A file that was extended and then written through a mapping needs both: the pages, so the contents are on media, and the inode, so the length on media matches. Without the second, recovery sees a file of the old length whose extended region was never logically part of it, and the bytes written there are lost even though they are physically present.

The ordering follows the same logic as the rename protocol. The data must be durable before the metadata that publishes it, because a length that points at unwritten bytes is a file that reads as garbage. So the sequence for appending through a mapping is: extend the file, map the new region, store the data, msync the data, then fsync the descriptor. The fsync is what makes the new length durable, and it does not imply the data’s durability on a mapping the kernel has not been told about, which is why both calls are needed.

There is a further subtlety with hole-punching and with sparse files. A page of a sparse file that was never written reads as zeroes without any block being allocated. A store through a mapping to such a page triggers a fault, allocates a block, and fills it. The allocation is a metadata change; the store is a data change. A crash between them can leave the block allocated and the data lost, or the data written and the allocation lost, depending on the order the filesystem chose. Filesystems that support this correctly order the allocation before the data becomes visible and flush in that order, which is a constraint the application cannot verify from outside.

Direct access removes the page cache

A filesystem mounted with direct access maps the media into the process’s address space without a page cache in between. A store to such a mapping writes to the DIMM through the CPU’s cache hierarchy, and a load reads from the DIMM through the same path. There is no dirty page, no writeback thread, and no msync needed to move data from a cache page to the media, because the mapping is the media.

What remains is the CPU cache, which is volatile, and the fence discipline described in Ordering Stores to Persistent Memory. A store through a direct-access mapping is durable when its cacheline has been written back and fenced, not when the store retires. This is why the instruction sequence and the mapping are used together: the mapping removes the kernel from the data path, and the flush instruction restores the durability guarantee that the kernel’s msync used to provide.

The gain is not only the avoided copy. A page-cache mapping of a persistent memory file has to move data from the page cache into the persistent region on writeback, which is a memory-to-memory copy of every byte written. A direct-access mapping has no such copy, so a write of N bytes costs N bytes of stores rather than N bytes of stores plus N bytes of copying plus the cache pollution of both. For a workload that writes a large region once, the difference is close to a factor of two in memory traffic, and the page-cache path also consumes page-cache memory that could hold something else.

The cost is that every access is a media access. A page-cache mapping gets the benefit of the cache for repeated reads; a direct-access mapping pays the DIMM’s latency every time, which is higher than an L3 hit but far lower than a device read. The crossover is a function of the access pattern, and it is why direct access is a mount option rather than a universal default.

Faults, alignment, and the granularity of a mapping

A mapping is established at page granularity, 4096 bytes on x86-64, and a fault on a page that has no block allocated costs a filesystem operation. On a direct-access mapping, the fault handler must allocate the block and program the mapping before the store can proceed, which means the first store to each page pays a cost the later ones do not. A program that writes sequentially through a fresh mapping therefore sees a fault per page, and a program that pre-faults the region with a read or an explicit touch pays them all upfront instead of interleaved with the work.

The mapping’s granularity also determines what msync can express. A range smaller than a page is rounded up, so two threads writing to different halves of the same page cannot be flushed independently, and a fence that orders one thread’s stores also orders the other’s as far as the media is concerned. This is not a correctness problem for a single writer, but it is a performance problem for a design that assumed page-level independence.

Unmapping has its own hazard. munmap does not flush; the pages remain dirty and are written back later. A program that unmaps and then reports success has reported the state of its address space, not the state of the media. The flush has to happen before the unmap, or through the descriptor afterwards, and the second path requires the descriptor to still be open.

What the mapping cannot express

A mapping is a linear view of a file’s bytes. It cannot express a write that must be atomic with respect to another file, a rename, a directory entry, or a link count. Those are namespace and metadata operations, and they go through the descriptor interface with its own durability rules, the same rules that make a flushed file still nameless if the directory was not flushed, as described in What fsync Actually Waits For.

A mapping also cannot express the ordering between two mappings of different files. Two msync calls, one per file, establish durability for each but not an order between them; a crash can leave one landed and the other not. If the two files are a data file and a log, the order matters, and the only way to get it is to make one call wait before issuing the other, which is exactly the barrier the application has to build itself.

The final limitation is the one that motivates the whole cluster: none of these interfaces can tell whether the device honored the flush. msync returns success when the kernel’s part is done. Whether the DIMM’s contents survive a power loss depends on the platform’s persistence domain and on the media’s own behavior, and no return value reports that. The only way to gain confidence is to test the failure, which is the subject of Verifying Crash Consistency with Fault Injection.