The Git Index: A Sorted Path Table Between Worktree and Commit

The working tree is a directory of ordinary files. The commit is an immutable graph of objects. Between them sits a third structure that is neither: .git/index, a single binary file listing every path that will go into the next commit, together with enough cached filesystem metadata to decide, without reading file contents, whether the working tree has diverged from it. The index is the only part of the repository core that is modified in place, and its format explains most of the behaviour of git status, git add, and git diff when they seem to disagree with each other.

Staging exists because a commit is built, not observed

A version control system that commits the working tree directly has no way to express “this change but not that one” within a single file. The index decouples the two operations: git add copies a blob into the object store and records its identifier under a path, while git commit converts the whole index into a tree. Committing is therefore a pure function of the index, and the working tree is irrelevant at that moment. A file modified after git add does not enter the commit.

The same structure holds the intermediate states of a merge. One path can have up to four entries in the index, distinguished by a two-bit stage number, and that is the mechanism by which a conflict is represented. Before the object graph can absorb a merge, the conflict has to be a data structure, and the index is where it lives.

The DIRC file layout

The file begins with the four bytes DIRC, a four-byte version number, and a four-byte count of entries. Versions 2 and 3 store entries in a fixed layout; version 4 replaces the literal path with a prefix-compressed encoding. After the entries come optional extensions, each introduced by a four-byte signature and a four-byte length, and the whole file ends with a SHA-1 checksum over everything preceding it.

An entry carries ten 32-bit values copied from the stat structure of the file it describes: ctime seconds and nanoseconds, mtime seconds and nanoseconds, device, inode, mode, uid, gid, and size. Then come the twenty bytes of the blob identifier and a 16-bit flags word. Twelve of those bits hold the path length, saturating at 0xFFF for longer paths, two bits hold the stage number, one marks an extended flags word as present, and one is the assume-valid bit. The path follows, NUL-terminated, and in versions 2 and 3 the entry is padded with NULs so that the next entry starts on a multiple of eight bytes.

Paths are stored sorted by byte comparison, with no locale or case folding. The ordering is load-bearing: it permits binary search for a path, and it makes the merge of two indexes a linear scan. It also explains why a checkout on a case-insensitive filesystem can produce an index whose order disagrees with the filesystem’s idea of identity, a mismatch Git reports as a duplicate-path error rather than silently resolving.

Reading the header is enough to see the structure:

import struct

with open(".git/index", "rb") as f:
    data = f.read()

signature, version, count = struct.unpack(">4sII", data[:12])
assert signature == b"DIRC"
print(version, count)          # 2, 3, or 4; count is the number of entries

# The entry table starts at offset 12. In versions 2 and 3 each entry is
# 62 bytes of fixed fields plus a NUL-terminated path padded to 8 bytes.

The stat cache and the racy timestamp

Status has to answer one question per path: does the file on disk still match the blob recorded for it? Hashing every file would make the operation proportional to the size of the working tree rather than to the number of files, which is unacceptable for a command run before every commit.

The stat cache answers it cheaply. If ctime, mtime, size, inode, and mode all match what the index recorded, the file is treated as unchanged and its contents are never read. Any mismatch causes a hash of the file and a comparison against the stored identifier; a hash mismatch marks the path as modified, and a match refreshes the cached stat data. This is why touching a file without changing it costs one hash but produces no diff.

The scheme has a hole. If a file is modified within the same timestamp tick as the index write, the recorded mtime and the file’s mtime are equal, and the stat comparison cannot distinguish “unchanged since staging” from “changed after staging”. Git calls this racy-git and handles it by treating entries whose mtime is not older than the index file’s own mtime as suspect, hashing them on every status until a later write makes the timestamps orderable. Filesystems with coarse timestamps make the window larger, and filesystems that report unreliable inode or device numbers make the cache useless; both can be worked around by configuration that narrows which fields are compared.

Two extensions reduce the cost further. The untracked cache records directory-level information so that a scan for untracked files can skip directories whose mtime has not changed. The filesystem monitor extension lets an external daemon report which paths changed, so status need not walk the tree at all. Both are derived data, and both are invalidated conservatively when anything about the directory changes.

Three entries for one path

A clean index has exactly one entry per path, at stage zero. During a merge, a conflicted path gains entries at stage one, two, and three, holding the blob from the merge base, from the current branch, and from the branch being merged. Paths that merged cleanly stay at stage zero.

The stage numbers are visible through git ls-files --stage, which prints mode, identifier, stage, and path. They matter because the tree-building step refuses to run while any nonzero stage exists: a commit cannot contain three versions of one path. Resolving a conflict is the act of writing a single stage-zero entry, either by choosing one side, by editing the file and adding it, or by removing the path.

The index also stores a resolve-undo extension, which remembers the three pre-merge identifiers after a conflict is resolved so that git checkout -m can reconstruct the conflict. This is separate from the object store, which never forgot the blobs; what the extension preserves is the association between those blobs and the conflicted path.

Extensions, split indexes, and sparse checkouts

The cache-tree extension stores the tree identifier of directories whose contents are known to match the index exactly. Without it, every commit would rebuild every tree from scratch, an operation proportional to the number of tracked paths. With it, a commit only rebuilds the directories containing changed entries, and the rest of the tree is reused by identifier. Any modification to an entry invalidates the cache for its directory and every ancestor.

A split index moves most entries into a separate shared file and leaves a small delta index behind, which makes repeated writes cheaper in repositories with hundreds of thousands of paths. A sparse index goes further and replaces whole directories with single entries carrying the skip-worktree bit, so that a checkout of a narrow subset of the tree does not need an entry per file outside it.

The skip-worktree and assume-unchanged bits are often confused. Assume-unchanged is a promise from the user that a file will not change, which lets status skip the stat comparison; Git is free to ignore the promise when it needs a correct answer. Skip-worktree is an instruction that the path is not present in the working tree by design, used by sparse checkout, and it is respected by checkout and status alike.

Where the format leaks

Because the index caches filesystem identity, it is machine-local and cannot be committed or shared; a fresh clone builds a new one. A crash during a write leaves a .git/index.lock behind and every subsequent write fails until it is removed. Checksum failures indicate truncation or a partially written file, and Git’s response is to refuse to operate rather than to guess. Conversion filters complicate the stat cache, since the blob stores the normalised form while the file holds the working-tree form, and a stat match on the file says nothing about whether normalisation would change the blob. And because entry ordering is bytewise, an index written on a case-insensitive filesystem can contain two paths that the filesystem considers identical, a state that survives until the next git add reports it.

The identifiers the index holds are ordinary blob objects, so staging is cheap: writing a blob is a hash and a compressed file, and the index entry is a pointer. Turning the index into a tree is the inverse, and the cache-tree extension is what keeps that operation proportional to the change rather than to the repository.

Nothing in the index records which branch it belongs to. That association lives in the reference namespace, where a single file holds the identifier of the commit the index is expected to descend from.