Layer Cache Invalidation in Container Image Builds

A container image build is a sequence of instructions, each producing a layer, and the cache is a chain rather than a set. That structural fact explains nearly every surprising cache behaviour: why editing a comment near the top of a file invalidates a dependency installation hundreds of instructions later, why the same instruction can hit on one machine and miss on another, and why reordering instructions changes nothing about the image and everything about the build time.

A layer is identified by its parent and its instruction

The cache key for a layer is a pair: the digest of the parent layer’s result, and the instruction that produced this layer, together with the content it consumed. Nothing else enters the key. This is what makes the chain property inevitable — a layer’s key contains its parent’s digest, so if the parent changes, every descendant’s key changes, and no descendant can hit.

The chain is a deliberate simplification with a real cost. It means the build cannot reuse layer N+1 when layer N misses, even if the instruction that produced N+1 is unrelated to the one that changed. A cache with per-instruction keys and no parent linkage could reuse more, but it could not guarantee that the filesystem state at the start of N+1 is the same as it was when N+1 was cached. The chain is what makes the guarantee possible: a hit implies the exact parent state that produced the cached layer.

For an instruction that copies files, the key includes a digest of the copied content. COPY . /app therefore invalidates whenever any file in the build context changes, including files the build does not use. That is the single largest source of unnecessary invalidation, and it is why build contexts are trimmed and why .dockerignore exists.

For an instruction that runs a command, the key includes the command string exactly as written. A change in whitespace, a reordered flag, a renamed variable inside the command text all invalidate. The builder does not parse the shell; it hashes the string. Two textually different commands that do the same thing are two different cache entries, and the second one has to run.

The instruction string is hashed, not interpreted

Consider the common pattern of installing dependencies before copying source. The ordering exists entirely because of the chain property: dependency installation depends on the manifest file, and source code changes far more often than the manifest, so the expensive install sits above the volatile COPY and stays cached across source edits.

COPY package.json package-lock.json /app/
RUN npm ci --omit=dev
COPY . /app
RUN npm run build

The fourth line copies everything, including node_modules if it exists in the context, which is why the ignore file must exclude it. The third line invalidates on any source change and correctly so, because the build step reads the source. The second line stays cached as long as the two manifest files and the base image are unchanged, which is the point of the arrangement.

Reverse the order — copy everything first, then install — and every source edit re-runs the install. The image is identical either way. The build time differs by the duration of the install, which for a large dependency tree is the dominant cost.

The same reasoning applies to package manager caches inside the build. npm ci and its equivalents delete or bypass any cache the image already had, so a build that relies on a warm package cache in a parent layer gains nothing. Builders that support cache mounts solve this by mounting a directory that persists across builds without becoming part of a layer, so the cache is available to the install and absent from the final image. That is a cache that does not participate in the layer chain at all, and it is the right tool when the chain is too coarse.

Base image changes propagate through every layer

The first instruction’s key includes the base image reference, and when that reference is a tag, the builder must resolve it to a digest to compare. Whether it does so automatically depends on the builder and its configuration; some resolve the tag on every build, some reuse a cached resolution, and some only resolve when explicitly told to pull.

The consequence when the resolution is stale is that the build reuses layers computed on top of an older base while reporting success against the newer tag. The layers are internally consistent — they were computed from that older base — so nothing fails, and the image is a mixture of the old base and new instructions that were written expecting the new base. Pinning the base by digest makes the situation explicit, using the same naming discipline as any other content-addressed artifact reference: either the digest is what the build expects, or the build says so.

A base image update also invalidates the whole chain, which is correct and expensive. The first instruction’s output digest changes, so the second’s key changes, and so on to the end. There is no way around this within the chain model, because the filesystem state at each step genuinely differs. Rebuilding everything after a base bump is the honest cost of changing the foundation.

Content digests versus file metadata

Builders differ in what they consider a change for a copied file. Content-only comparison treats a file as unchanged if its bytes are unchanged, regardless of modification time or permissions. Metadata-aware comparison includes the mode, and sometimes the owner, in the digest.

The difference shows up in two directions. A build that only changes a file’s permission bits will invalidate under metadata-aware comparison and hit under content-only, and the metadata-aware behaviour is the correct one for a copy that preserves mode, because the resulting layer’s metadata genuinely differs. A build after a fresh checkout may see every file’s mtime change while the content is identical; mtime is not part of a layer’s content, so a builder that keys on mtime invalidates everything for no reason.

Permissions are a common source of this. A git clone on a filesystem without executable-bit support, or a checkout through an archive that does not preserve modes, produces files whose mode differs from the developer’s machine. The copy then produces a different layer, the chain breaks, and the difference is invisible in the file contents.

Cache export, import, and the registry round trip

A local cache is useless in a CI system where each build starts on a fresh machine. The builder can export cache metadata to a registry or to a file, and a later build on a different machine can import it. The exported cache contains the layer keys and the layer blobs, or references to them.

Two properties of this mechanism matter. First, the export is keyed the same way as the local cache, so an imported cache hits under exactly the conditions a local one would. It does not make an under-keyed cache safe; it only makes it portable. Second, the export has a size, and for a large dependency tree the cache can be larger than the image. Storing it in the registry costs storage and transfer on every build that pulls it, which can exceed the cost of rebuilding when the build is cheap.

The usual compromise is to export cache only for the stages whose rebuild is expensive, and to let the rest rebuild. That requires knowing which stages those are, which is a measurement rather than a principle, and the measurement has to be redone when the dependency tree changes shape.

Multi-stage builds interact with this in a useful way. A stage that produces a build toolchain can be cached and exported on its own, and the final stage can copy from it without inheriting its layers. The final image then contains only the artifacts, not the toolchain, which is the standard reason for multi-stage builds and also the reason their caches are worth exporting separately.

When the cache is wrong

The cache can return a layer that does not correspond to the current instruction, and the mechanisms are specific.

A command that reads the network produces a layer that depends on what the network served. The instruction string is unchanged, the parent is unchanged, and the cache hits, returning a layer built from an older version of whatever was downloaded. Package installs that do not pin versions are the canonical case: the manifest is unchanged, the registry’s contents are not, and the cache returns the old resolution. The cache is doing exactly what it was told; the build’s inputs are incomplete.

A command that reads the clock behaves the same way, and so does one that reads a file outside the context. All three are instances of the under-keying problem described in What a Build Cache Key Must Cover, expressed through a mechanism that hashes instructions rather than declaring inputs.

The remedies are the ones that apply generally. Pin the versions that matter so the network is not an input. Mount a cache instead of reading a network service where the tool allows it. Set the timestamps the build reads. And when a build must see fresh remote state, invalidate deliberately by changing something in the instruction, since that is the only lever the chain model offers.

The chain also has a failure mode in the other direction: a cache that never hits because something above the expensive step changes on every build. A COPY of a file that contains a build number, an ARG whose default includes a timestamp, an instruction that installs a package from a repository whose metadata changes — any of these at the top of the file invalidates everything below. Finding them is a matter of reading the build’s own cache report, which most builders can produce, and looking for the first instruction that misses on consecutive builds with no source change.