Doublewrite Buffers and the Torn Page Problem

Write-ahead logging rests on an assumption that is false at the hardware level: that writing a page either happens entirely or not at all. A disk write is a transfer of a buffer over a period of time. Power loss, a device reset, or a bus error during that transfer leaves some sectors with the new contents and some with the old. The result is a page that is neither version — a torn page — and no amount of log replay can fix it, because the log describes the change from the old version to the new one, not how to repair a mixture of the two.

Why recovery cannot repair a torn page

Redo is idempotent because it is guarded by the page LSN. A record is applied only if the page’s stored LSN is less than the record’s. A torn page has a header from one version and a body from the other, so its LSN may read as either. If it reads as the new LSN, redo skips the page entirely and the mixed contents survive. If it reads as the old LSN, redo applies the change to a body that is partly new, and the arithmetic in the record is applied twice to the parts that were already updated.

Neither outcome is detectable from the log alone. The engine must therefore be able to recognize a torn page and to reconstruct it from something other than the log. The two mechanisms that do this are checksums and a second copy.

Detection: the page checksum

The simplest defence is a checksum over the page contents, stored in the header and verified on every read. A torn page almost always fails the checksum, because the boundary between old and new sectors falls somewhere in the checked region. PostgreSQL stores a 16-bit checksum in the page header and computes it over the whole block; a mismatch raises a read error naming the block rather than silently returning corrupt tuples.

Checksums detect but do not repair. Their value is that they convert silent corruption into an error, and that they make the second mechanism’s decision cheap: an engine that finds a bad checksum on a page knows to look up the shadow copy. A checksum also catches corruption that is not torn writes — a failing sector, a bit flip in memory, a bug in the write path — which is a large fraction of the practical value.

The weakness is that a 16-bit checksum has a small probability of collision: roughly one in 65,536 for random damage, and lower for the structured damage a torn write produces. That is good enough to detect the common case and not a guarantee. Some engines use a 32-bit CRC for a correspondingly smaller collision rate.

The shadow copy

The doublewrite buffer is a region of storage — in InnoDB, a set of contiguous extents inside the system tablespace — to which pages are written before they are written to their final location. The write sequence is:

1. Write the page to the doublewrite buffer, sequentially, several pages at a time.
2. Flush the doublewrite region to stable storage.
3. Write the page to its final location.
4. Mark the doublewrite slot reusable once the final write is durable.

If a crash occurs during step 3, the page at the final location may be torn, but the doublewrite copy is intact — step 2 guaranteed that. Recovery reads the doublewrite region, finds pages whose final copy fails its checksum, and restores them from the shadow copy. Then normal redo applies.

The important structural property is that step 1 is a sequential write. Pages that are dirty are scattered across the buffer pool and their final locations are scattered across the tablespace; writing them directly produces random I/O. Writing them first to a contiguous region batches the writes and turns the first pass into a sequential one. On rotating media that is a genuine win, and it is why the doublewrite buffer survived as an optimization rather than only as a correctness measure. On flash the sequential advantage is smaller, and engines offer the option to disable it — which is safe only if the device guarantees atomic page writes, a property some devices advertise and few guarantee across a power failure.

/* The write path with a shadow copy; ordering is the whole mechanism. */
int flush_page(page_t *p) {
    if (doublewrite_enabled) {
        slot_t *s = dw_reserve();                 /* sequential append */
        if (!s) return -EAGAIN;
        memcpy(s->data, p->data, PAGE_SIZE);
        if (dw_flush_to_stable(s) != 0) return -EIO;
        p->dw_slot = s->id;
    }
    if (pwrite(fd, p->data, PAGE_SIZE, p->offset) != PAGE_SIZE)
        return -EIO;                              /* may be torn; shadow still valid */
    if (doublewrite_enabled) dw_release_after_fsync(p->dw_slot);
    return 0;
}

The release after the final write must itself be ordered: a slot cannot be reused until the write it shadows is durable, or a crash could find the slot overwritten by a newer page while the older page is torn. InnoDB batches this by flushing the doublewrite region, writing the pages, then flushing again before advancing the reusable boundary.

The cost, in bytes and in latency

Every page write becomes two writes, one sequential and one at the final location. That doubles the bytes written per page flush, and it adds one flush boundary per batch. For a workload already writing two bytes per user byte — log plus page — the doublewrite pushes the total to three.

The latency cost is concentrated in the flush boundaries. Each batch requires waiting for the shadow copy to reach stable storage before the final writes may begin, so the batch’s throughput is bounded by the flush rate rather than by the device’s raw write bandwidth. Larger batches amortize the flush; smaller batches reduce the window during which a slot is held.

There is a space cost too: the doublewrite region is a fixed allocation, typically a small percentage of the tablespace, and it is a serialization point — all pages being flushed compete for slots in it. A pool with many dirty pages and a small doublewrite region will find the flush path throttled by slot availability rather than by I/O capacity.

Alternatives and their trade-offs

Full-page writes in the log are the alternative used by PostgreSQL. Instead of a shadow copy area, the first modification of a page after a checkpoint logs the entire page image, so recovery can restore the page from the log. This costs log space proportional to the number of pages modified per checkpoint, but it costs nothing for pages modified repeatedly within one checkpoint interval, because only the first modification needs the full image. The trade is between a fixed per-page-write doubling and a log volume that depends on how many distinct pages a checkpoint interval touches.

A third approach is to make pages self-verifying and to rely on redundancy at a higher level — checksums plus a replica — accepting that a torn page on the primary is repaired by copying from a replica. That works only when a replica exists and is itself consistent, and it converts a local recovery problem into a network operation.

A fourth is hardware: a device that guarantees atomic writes of a full sector, or of a full page with a power-loss-protected cache, removes the need for the shadow copy. The guarantee must extend across power loss, not merely across a clean shutdown, and it must cover the page size the engine uses rather than the sector size the device reports.

Interaction with checksums and with redo

The doublewrite buffer and the checksum are complementary and neither is sufficient alone. The checksum identifies which pages need repair; the shadow copy supplies the repair. Without a checksum, recovery cannot tell a torn page from a valid one and would restore every page in the doublewrite region, which is wasteful but not incorrect — provided the region’s contents are still the ones that were written, which is why the region must not be reused prematurely.

Without a shadow copy, a checksum gives only detection: the engine must then fail the read, and the operator must restore from a backup. That is a defensible position for an engine that expects to run on storage with atomic writes, and an indefensible one for an engine that expects commodity disks.

The redo interaction is that restored pages carry the LSN from the shadow copy, which is the LSN of the version that was written, and redo then proceeds from there. A page restored to a version older than the checkpoint’s redo point will be brought forward by redo; a page restored to the newest version will be skipped. Both are correct because the LSN comparison is what decides, which is the same mechanism described in write-ahead log record ordering.

Where it shows up in operation

The doublewrite buffer’s presence is visible in three ways. Bulk loads that bypass the buffer pool and write pages directly — some engines offer this for newly allocated extents — skip it, on the grounds that a torn page in a region with no live data can simply be discarded. Checksum failures reported after a crash are followed by a restore from the shadow copy and a log entry; failures reported without a preceding crash indicate media damage and are not repaired by the buffer.

And the region’s size is a tuning parameter with a sharp failure mode: too small and the flush path serializes on slot allocation, too large and it wastes space that could hold data. The correct size follows from the number of pages the engine intends to flush per batch, not from the tablespace size.

The page format that the checksum covers, and the header fields it must exclude, are described in slotted page layout. Whether the second write can be avoided by never modifying pages in place is the question that LSM compaction strategies answers differently.