Crash-Consistent and Application-Consistent Backups

The appeal of a snapshot is that it appears to require no cooperation from anything above the storage layer. A volume is copied at an instant, the copy is stored somewhere else, and the application never knows. The appearance is misleading in a precise way: the snapshot is a true image of the block device at one instant, but the block device is not the application. Writes that the application has issued but that have not reached the device are not in the image, and writes that have reached the device but that the application has not finished grouping into a logical operation are in the image without the rest of their group.

Those two problems have different fixes, and confusing them produces backups that restore cleanly and lose data, or that contain every byte and cannot be mounted.

What the snapshot actually captures

A snapshot is a metadata operation. On a copy-on-write filesystem it records the current root of the block tree and arranges for later writes to allocate new blocks rather than overwrite the snapshotted ones. On a storage array it does something equivalent inside the controller. In both cases the operation itself is fast because no data is copied, and the snapshot’s contents are the state of the device as of the moment the metadata switch happened.

Everything written before that moment and successfully persisted is in the image. Everything still sitting in a cache is not. This is exactly the state a power failure at that instant would have left behind, which is why the result is called crash-consistent: it is the state the machine would recover from after an unexpected loss of power.

That definition is worth taking literally, because it determines what recovery will involve. A crash-consistent image of a filesystem mounts only after journal replay, and replay restores the filesystem’s own invariants: no block claimed by two files, no directory entry pointing at a freed inode, no half-updated extent tree. It does not restore any invariant that lives above the filesystem.

The three caches between the write and the disk

Between an application’s write and the platter or flash there are at least three places the data can wait.

The first is the page cache. A write returns as soon as the bytes are in kernel memory. The filesystem’s writeback flushes dirty pages later, on a timer or under memory pressure. A snapshot taken while pages are dirty captures the device state without them, which means the image can be older than writes the application considered complete. For an application that never called fsync, that is expected and unavoidable. For one that did, the guarantee is only as good as what fsync promises.

POSIX requires fsync to transfer all modified data and metadata for a file to the underlying storage before returning, and to report an error if it cannot. fdatasync transfers the data and only that metadata needed to retrieve it, which excludes timestamps but includes the file size. Neither call flushes the directory entry that names the file, so a newly created file is not guaranteed to survive a crash unless the containing directory is also synced. A program that creates a file, writes it, and calls fsync on the file descriptor has not made the name durable.

The interval between an acknowledged write and the image that contains it is a data-loss window, and it is the same quantity that log shipping and the recovery point objective quantifies for transaction logs. A snapshot schedule of one hour cannot promise a better recovery point than one hour, regardless of how quickly the snapshot itself completes.

The second cache is the storage controller’s. A RAID controller or SAN array acknowledges writes once they are in its battery-backed or capacitor-backed memory, and writes them to the actual media later. A snapshot taken by the array itself orders correctly with respect to that cache, because it is the same device. A snapshot taken by a hypervisor, or by a filesystem that sits above a network device, may not. In that case the cache must be flushed before the snapshot, which is a SCSI SYNCHRONIZE CACHE command rather than a filesystem operation. Filesystem-level fsync does not necessarily issue it.

The third cache is inside the drive. Consumer drives may reorder writes within their volatile buffer unless the write is issued with a force-unit-access flag; the filesystem’s barrier and flush operations exist to produce that flag. Disabling barriers for performance removes the ordering that crash consistency depends on, and does so invisibly: the filesystem still mounts after a crash, but the journal may describe a state that never existed.

Quiescing without stopping the service

Application consistency means the image contains a state the application can start from, not merely one the filesystem can mount. Two files that the application updates in sequence must both reflect the same transaction, and a database’s data files must correspond to a point in its log.

The strongest method is to stop the service, let it flush, take the snapshot, and start it again. That is also the method with the largest outage, and for a database it discards the buffer pool, so the restart is slow and cold.

The middle method is a filesystem freeze. On Linux, the FIFREEZE ioctl flushes the filesystem and blocks new writes from entering it; FITHAW releases it. The freeze is per-filesystem, and it blocks writers rather than failing them, so an application blocked during the freeze simply stalls. The snapshot needs to cover only the interval during which the filesystem is frozen; with a copy-on-write snapshot that interval is metadata-only and short, while an array snapshot that must quiesce the volume may hold the freeze for seconds. The outage scales with the snapshot mechanism, not with the volume size.

The third method is cooperation from the application, and it is the only one that produces a state the application itself understands. A database that supports online backups writes a marker into its log, records the start position, lets the filesystem copy proceed, and then records the end position. On restore, the log is replayed from the start marker to a chosen point. PostgreSQL’s protocol is the canonical example: the backup begins with a function that forces a checkpoint and returns the starting WAL location, the files are copied while the server keeps running, and the backup ends by recording the stop location. Recent versions renamed these functions and changed the mechanics of the backup label file, so a procedure written against one major version does not necessarily apply to another. The important property is not the function name but the ordering: the copy is bracketed by log positions, so the copy does not need to be atomic.

The weakness of the cooperative method is that it depends on the application noticing. A database whose backup mode was entered and never exited keeps its log from being recycled, and the log directory grows until the volume fills. A backup agent that crashes between the start and stop markers leaves that state behind, and the next backup inherits it.

Why a torn page is detectable but a torn transaction is not

Storage guarantees atomicity at the sector, not at the page. A 512-byte sector write either happens or does not; a write spanning several sectors, issued without a force-unit-access flag, can be partially applied. Database pages are larger than sectors, typically 8 KiB or 16 KiB, so a crash during a page write leaves a page that is half old and half new. Neither half is corrupt on its own, and the result is a page that passes no checksum: most engines store a checksum per page precisely to detect this.

The standard repair is to write the full page into the log at the first modification after each checkpoint, so that replay can restore a known-good copy before applying the change. PostgreSQL calls this full-page writes; the equivalent idea appears in most write-ahead-logged engines. The cost is log volume proportional to the number of pages dirtied after a checkpoint, which is why the setting trades write amplification against the ability to recover from torn pages.

A torn transaction has no equivalent checksum. If an application writes message contents to one file and updates an index in another, a crash-consistent image can contain the message without the index entry or the index entry without the message. Nothing in either file is corrupt; the pair is inconsistent. Recovery code may or may not notice, and if it notices, it may or may not repair in the direction that preserves data. This is the class of damage that filesystem-level consistency cannot address, and it is why an image taken without quiescing is a backup of a filesystem rather than a backup of a service.

Choosing the weaker guarantee on purpose

Crash consistency is not a failure of the backup system. It is the correct and only guarantee available for a workload that cannot be quiesced: a laptop, a busy multi-tenant hypervisor, a volume holding data from a dozen services with no single owner. The decision is whether the restore procedure accounts for it.

An image that is only crash-consistent requires a recovery step after restore, not just a mount. The application’s own recovery must run and must be allowed to fail visibly. A restore procedure that mounts the image, starts the service, and checks that the process is running will pass on a database that has silently discarded its uncommitted tail and may pass on one whose indexes no longer match its rows.

The stronger guarantee costs a write stall whose length is set by the snapshot mechanism and a dependency on the application’s cooperation being correctly bracketed. The weaker one costs an unbounded amount of recovery work at the worst possible time. Both are defensible; what is not defensible is believing the image has the stronger property because the snapshot command returned successfully.

Whether the resulting image can be restored at all is a separate question from whether it is consistent, and it depends on the shape of the backup chain described in incremental backup chains and restore depth.