End-to-End Protection Information and Where the Checksum Stops

Every checksum discussed so far is computed by the host and verified by the host. Between the two events the data passes through a device driver, a host bus adapter, a cable or a fabric, a drive’s controller, its cache, and its medium. Each of those stages can corrupt the bytes, and a checksum that is only checked at the far end of the journey cannot say which stage did it — or, if the corruption happens after the last host-side check and before the medium, cannot detect it at all.

Protection information is the storage industry’s answer: an 8-byte field carried alongside each logical block, written by the host, checked and regenerated by every stage that handles the block, and verified by the host on read. It is the closest thing storage has to an end-to-end check, and its limits are as instructive as its coverage.

The eight bytes and what each defends

The format is fixed by the SCSI Block Commands standard and reused by NVMe. Each logical block carries:

A guard field, 16 bits, holding a CRC computed over the block’s data plus the other two tag fields. It detects corruption of the data or the tags anywhere the block was stored or transported.

A application tag, 16 bits, whose meaning is defined by the user of the protection information. Filesystems and volume managers use it to carry an identifier for the owning object, so that a block arriving in the wrong context is rejected even if its address is right.

A reference tag, 32 bits, holding the low 32 bits of the logical block address the block is supposed to occupy. This is the field that catches misdirected writes, and it is the reason the scheme exists at all.

The reference tag is the elegant part. A write to LBA 0x1A2B carries a reference tag of 0x1A2B in its protection information. The drive checks, on receipt, that the tag matches the address the command names — a mismatch is a command error, caught before anything is written. On a subsequent read, the drive regenerates the tag from the address being read and checks the guard against the stored data; if a block was written to the wrong address by some path that bypassed the check, the tag it carries will not match the address it now occupies, and the read fails. This is precisely the failure that a host-computed checksum stored in metadata cannot catch, because the metadata checksum describes content, not location.

Protection information comes in three types, differing in how much of the tag is enforced. Type 1 checks the guard and the reference tag and leaves the application tag to the application. Type 2 checks all three, which is the mode a filesystem or volume manager uses when it has something meaningful to put in the application tag. Type 3 checks only the guard and ignores both tags, which is useful for transports that cannot carry tags end to end. The type is negotiated between host and device; a device that supports only type 3 provides media protection but not misdirected-write protection.

Where it sits in the stack

The two placements are known as DIF and DIX, and the difference is where the 8 bytes live.

In the interleaved form, the protection information is appended to each logical block in the same buffer that carries the data. A 512-byte block becomes 520 bytes on the wire and on the medium; a 4096-byte block becomes 4104. The overhead is 1.6 percent for 512-byte blocks and 0.2 percent for 4096-byte blocks, which is one reason the industry’s move to 4 KiB sectors made protection information cheaper to adopt. The advantage is that the protection travels with the data through every stage without any separate bookkeeping; the disadvantage is that the medium must be formatted for it, which means the capacity is gone whether or not the host uses it.

In the separate form, the data occupies a normal block and the protection information is transferred in a parallel buffer, either in a separate memory region for a DIX-style transfer or in a separate metadata region on the medium for NVMe. The host can then compute protection information only when it wants to, and the device stores it in a metadata area whose size is configured independently. The advantage is flexibility and no capacity loss on media that do not need it; the disadvantage is that the protection information is a second stream that must be kept in step with the data, and a device that drops or misroutes one stream defeats the check.

NVMe carries the same 8-byte structure in its metadata, with the same three types and the same guard, application tag, and reference tag semantics. The transport differs — metadata can be interleaved with data in the same submission or transferred as a separate buffer — but the protection model is the same one SCSI defined.

What the host has to do

Protection information only works if the host actually computes it, and computing it means the host knows the LBA of every block it writes. That is a stronger requirement than it sounds, because a filesystem’s write path often does not: it hands a buffer and a file offset to the block layer, and the mapping to physical addresses happens later, in the allocator or the RAID layer.

The usual architecture is therefore a layer that sits between the filesystem and the device and owns the LBA mapping: a volume manager or a device-mapper target. It takes the write, computes the guard over the data and tags, sets the reference tag to the physical block address, and passes the combined buffer down. On read it verifies, and on a mismatch it can consult redundancy — which is exactly the self-healing logic of self-healing reads and reconstructing a damaged copy, applied one layer lower and using parity or a mirror rather than a filesystem’s block pointer.

The application tag is where a filesystem can add value if it participates. A value derived from the owning inode or extent lets the layer distinguish “this block is at the wrong address” from “this block belongs to a different file entirely”, which is a finer diagnosis than the reference tag alone provides. It also lets a snapshot or clone share blocks without the protection information becoming ambiguous, provided the tag is derived from something stable across the sharing.

/* Build protection information for one 4 KiB block. */
#include <stdint.h>
#include <string.h>

struct pi { uint16_t guard; uint16_t app; uint32_t ref; };

/* CRC-16 over data followed by the tag fields, with the polynomial the
   standard fixes for the guard field. */
uint16_t crc16_t10(const void *data, size_t len, uint16_t app, uint32_t ref);

void make_pi(struct pi *pi, const uint8_t *data, size_t len,
             uint16_t app, uint32_t lba)
{
    pi->app = app;
    pi->ref = lba;
    /* Guard covers data followed by the tag fields, in that order. */
    pi->guard = crc16_t10(data, len, app, lba);
}

int check_pi(const struct pi *pi, const uint8_t *data, size_t len, uint32_t lba)
{
    if (pi->ref != lba)
        return -1;                       /* block is not where it claims */
    return crc16_t10(data, len, pi->app, pi->ref) == pi->guard ? 0 : -2;
}

The order of the guard computation matters and is fixed by the standard: the CRC covers the data bytes first and the tag fields after, so a device that recomputes the guard from the stored data and tags gets the same value the host produced. An implementation that covers only the data produces guards that the device rejects, which surfaces as a command failure on every write rather than as a subtle integrity problem.

The boundary the scheme does not cross

Protection information closes the gap between the host’s buffer and the medium. It does not close the gap between the application’s buffer and the host’s buffer. A page of data that is corrupted in memory before the filesystem sees it, or after the filesystem returns it, is invisible to every check described so far, because the corruption happened above the layer that computes the guard.

That is a real boundary, not a technicality. The filesystem’s checksum protects against damage after the write and before the read; protection information protects against misdirection and transport corruption on the way to and from the medium; neither protects the copy that lives in the page cache, the copy that a direct-memory-access engine moves into the device, or the copy the application reads out of its own structures. Covering those requires a check computed at the application’s boundary, over the same bytes the application will use, and verified by whoever consumes them — which is what an application-level checksum or a database page checksum provides.

The layering is therefore three deep, and each layer covers faults the others cannot: application checksums cover memory and software faults, filesystem checksums cover address-independent content damage on the medium, and protection information covers misdirected writes and transport faults. The reason all three are worth their cost is that the faults they address are genuinely disjoint, which is the end-to-end argument applied concretely rather than as a slogan.

Cost and deployment reality

The measurable costs are small but not zero. The guard computation adds a CRC over each block on the write path and again on the read path; the 16-bit CRC is expensive enough in software that controllers normally compute it in hardware, and the Data Integrity Extensions permit a lighter checksum when the host supplies the metadata. The reference tag comparison is a 32-bit compare. The application tag is a register write. The storage cost is the 8 bytes per block, which at 4 KiB granularity is 0.2 percent of capacity.

The real cost is compatibility. A device must be formatted with protection information enabled, which is a destructive operation; a RAID controller must be configured to pass it through rather than strip it; a volume manager must be in the path to compute it; and the filesystem must tolerate the extra bytes in its block size arithmetic. Each of those is a place where the feature can be silently disabled — a controller that strips protection information produces a device that reports support and delivers none of the benefit, and the host has no way to tell from a successful write.

Verification is also asymmetric in a way that matters for diagnosis. A guard mismatch on read tells the host that the block is damaged but not where; a reference tag mismatch tells the host that the block is at the wrong address, which is a much more specific and much more alarming finding, because it implies a write path fault rather than media decay. Reporting the two differently is worth the effort: the first is a disk to be replaced, the second is a bug or a misconfiguration to be found.

The residual exposure is the one every checksum scheme shares. A write that is acknowledged but never reaches the medium leaves the previous contents in place, and the previous contents carry valid protection information for their own address. No check on the block can detect the absence of an update. That class is addressed by ordering and cache-flush semantics rather than by checksums, and it is one of the blind spots that checksum blind spots and misdirected writes catalogues.