Write-Ahead Journaling and the Three-Phase Commit Record

A journal is a small append-only region of the filesystem used as a staging area for metadata changes that will later be written to their permanent homes. The idea is borrowed from databases: if the intent to change something is durable before the change itself is attempted, then a crash can always be resolved by finishing or discarding the recorded intent. What makes the filesystem version interesting is that the unit of intent is a set of raw blocks, not a set of row updates, and the recovery procedure must be safe against being interrupted itself.

The mechanism in wide use, the journaling block device shared by ext3 and ext4, with structurally similar designs in XFS and JFS, organizes each transaction into three parts: a descriptor that lists where the pending blocks belong, the blocks themselves, and a commit record that declares the transaction complete. Only the last of these is the atomicity point.

Why the intent must precede the effect

Consider the append from the crash-window analysis: a bitmap update, a data block, and an inode update. Under write-ahead logging these become a transaction whose descriptor lists three target blocks, the bitmap block, the data block, and the inode block, each with its new content. The descriptor and the three payload blocks are appended to the journal, and then a barrier forces them to media. Only after that barrier completes is the commit record written.

If the power fails before the commit record is durable, the journal contains a transaction without a terminator. Recovery scans forward, finds a descriptor whose sequence of payload blocks is incomplete or whose commit record is absent, and treats everything from that point as never having happened. The in-place structures were never touched, so the filesystem is exactly as it was before the operation began. If the power fails after the commit record is durable, recovery replays the listed blocks to their home locations, and the operation completes.

The ordering constraint is strict in one direction only. Payload blocks must be durable before the commit record; the commit record must be durable before any in-place write begins. A filesystem that starts applying in-place writes before the commit record is durable has reintroduced the original problem, because a crash can now leave a partially applied transaction whose intent was never recorded. Some implementations relax this by allowing checkpointing to begin as soon as the commit record is issued rather than confirmed, relying on the fact that the journal still contains a complete copy, but the payload-before-commit ordering is never relaxed.

The anatomy of a transaction

A descriptor block carries a magic number that identifies it as journal metadata, a block type, a sequence number identifying the transaction, and a list of tag entries. Each tag names a target block number in the filesystem and a flag word; the flags record bookkeeping such as whether a data block’s first four bytes happened to equal the journal’s magic number, in which case the block is escaped so that recovery does not mistake it for journal metadata. The payload blocks follow the descriptor in the order the tags listed them, so the n-th payload block belongs at the n-th target named by the descriptor.

The commit block repeats the magic and the sequence number. Formats that add journal checksums record a checksum over the transaction, either as a single value covering the descriptor and payload blocks or as per-block checksums carried in the descriptor’s tags, so recovery can detect a transaction whose blocks were written but whose contents were damaged in transit, a case that a bare “commit block exists” test would accept and replay.

Sequence numbers matter because the journal is a ring buffer of fixed size, not an infinite log. When the journal wraps, the new transaction’s blocks overwrite the oldest ones, and the only way to tell a valid transaction from a stale remnant at the same physical offset is the 32-bit sequence number. Recovery reads the journal’s superblock to learn where the valid region begins, then walks forward checking that each descriptor’s sequence number is the expected next value. A descriptor whose sequence number is out of order means the region beyond it has already been reused, and scanning stops. With 32-bit sequence numbers and a journal that holds a few thousand blocks, the wrap distance is enormous; the number is not a correctness limit in practice but it does mean a filesystem image that has been reused or restored from an older snapshot can present a journal whose sequence numbers look plausible while describing stale content.

A simplified replay loop, stripped of the ext4-specific tag decoding:

struct jbd2_header {
    uint32_t magic;      /* 0xc03b3998 identifies journal metadata */
    uint32_t blocktype;  /* 1 descriptor, 2 commit */
    uint32_t sequence;
};

/* Returns the offset to resume scanning from, or 0 if the journal is clean. */
static int replay(struct journal *j, uint64_t start, uint32_t expect_seq)
{
    for (;;) {
        struct jbd2_header h;
        if (!read_journal_block(j, start, &h))
            return 0;
        if (h.magic != 0xc03b3998 || h.blocktype != 1 || h.sequence != expect_seq)
            return 0;            /* stale or truncated: nothing further is valid */
        int ntags = decode_tags(j, start, &h);
        if (ntags < 0)
            return 0;
        uint64_t commit = start + 1 + ntags;
        struct jbd2_header c;
        if (!read_journal_block(j, commit, &c) || c.blocktype != 2 || c.sequence != expect_seq)
            return 0;            /* payload durable, commit not: discard the transaction */
        if (!verify_checksum(j, start, commit))
            return 0;            /* contents damaged: never replay a partial copy */
        apply_payloads(j, start, ntags);
        start = commit + 1;
        expect_seq++;
    }
}

The loop’s structure encodes the entire contract. It never applies anything before verifying that a commit record for the same sequence exists, and it never continues past the first inconsistency, because a valid transaction followed by a gap followed by another valid transaction cannot occur in a correctly written journal. The gap means the ring wrapped and the later blocks are from an older generation.

Replay must be idempotent

Recovery can itself be interrupted. A machine that crashes during replay, reboots, and replays again must converge on the same result. This forces the payload writes to be absolute rather than relative: each journal block contains the complete new content of a target block, so writing it twice produces the same bytes as writing it once. No transaction may record “increment this counter” or “append to this list” as its effect, because replay would apply the increment repeatedly.

Idempotence also constrains how the journal tracks progress. Replay can begin at the journal’s start pointer and apply transactions in order, then update the start pointer to the end of the last replayed transaction. If the update to the start pointer is lost, the next replay re-applies the same transactions, which is harmless precisely because they are absolute. The start pointer’s update is therefore a performance optimization rather than a correctness requirement, and implementations update it lazily.

The one structure where idempotence is not automatic is the free-space accounting. Replaying a bitmap write that marks a block allocated is idempotent, but if the transaction also freed a block whose bitmap write was recorded in an earlier transaction that has already been checkpointed, the order of application matters. This is the problem that revoke records solve: a transaction that frees a block records a revocation in the journal so that recovery skips any earlier, still-uncheckpointed journal copy of that block’s metadata. Without revocations, replaying a chain of transactions in sequence would apply them in the wrong order relative to the freed block’s reuse.

What the journal costs

Every metadata block is written twice: once to the journal and once to its final location. For a metadata-heavy workload such as creating and deleting many small files or updating directories, the extra write traffic is close to 2x on the metadata path, and the barrier per transaction converts what would be independent writes into a serialized dependency chain. Batching is the mitigation: a transaction accumulates changes for a period before committing, so the barrier cost is amortized over many operations. The tension is that a longer batching window increases the amount of work lost on a crash and the amount of journal space needed to hold uncheckpointed transactions.

Journal size therefore bounds how far checkpointing may lag. If the journal is full of transactions that have been committed but whose in-place writes have not completed, new transactions cannot be appended and the filesystem must stall until checkpointing drains. This is the mechanism behind the stalls that appear under sustained metadata write load, and it is the reason journal sizing is a capacity question rather than a durability question.

The alternative strategies avoid the double write entirely. Copy-on-write filesystems write new blocks once and publish them through a new root pointer, so there is no journal and no replay. What they give up is in-place update: every metadata change costs a path of new blocks rather than one. Ordered metadata updates keep the in-place layout and use dependency tracking instead of a log, which removes the double write but requires reasoning about every pair of structures that an operation touches. The journal’s appeal is that it makes that reasoning local: whatever the transaction contains, the recovery procedure is the same.