Content-Addressed Cache Keys for Incremental Asset Builds

Conditioning an asset library is expensive in a way that scales badly. A source texture may need resizing, channel packing, mip generation, and block compression; a source mesh may need welding, tangent generation, quantization, and atlas packing. The outputs are what the runtime consumes, so the build must run whenever an output could differ from what is on disk. The engineering problem is deciding “could differ” cheaply and correctly, because the two failure directions have very different costs: a missed rebuild ships a stale asset, and an unnecessary rebuild costs minutes of machine time.

Timestamps and why they fail

The traditional answer is a modification-time comparison: if the input’s mtime is newer than the output’s, rebuild. It is cheap, it requires no hashing, and it is wrong in several ways that all occur in practice.

A timestamp records when a file was last written, not what it contains. Checking out a branch rewrites every file and updates every timestamp without changing any content, so the build reruns entirely. Copying a directory can preserve or discard timestamps depending on the tool’s flags. Restoring from a backup sets mtimes to the restore time. And the reverse error is worse: a file whose content changed but whose mtime was preserved — a script that writes in place, an editor configured to keep timestamps, a touch to a fixed date — is skipped silently, and the stale output survives.

There is also a granularity problem. Filesystems record mtime at second or sub-second resolution depending on the platform, and a build that reads an input and writes an output within the same tick produces an output whose mtime is not newer than its input’s. The next build then reruns. Some build tools compensate by comparing equal timestamps as “out of date”, which converts the race into a permanent rebuild for that target.

Content addressing replaces the timestamp with a function of the bytes. The input’s identity is a cryptographic digest of its contents, so the comparison becomes “is the recorded digest of the inputs equal to the current digest” — a question whose answer is independent of when or how the bytes arrived.

What the key must contain

A cache key that hashes only the source file is wrong in a way that is easy to miss, because it produces correct results until the first settings change and then silently produces stale ones forever. The key must cover everything the output depends on:

• The input bytes. Every source file the operation reads, including files reached indirectly, such as an include, a referenced texture, or a table the tool consults.
• The tool’s identity. Not just the name, but the version. An encoder that changes its block-selection heuristic produces different bytes from identical input, and the version string is the cheap proxy for that. A tool that reports only a semantic version, and changes behaviour within a patch release, defeats this.
• The parameters. Every setting that affects output: compression format, quality level, mip filter, colour space tag, quantization bits. A pipeline that stores these in a project file hashes the project file; one that stores them in per-asset metadata hashes the metadata.
• The dependency’s own key. For a chained operation such as source image to packed atlas to compressed texture, the downstream key must include the upstream key, not the upstream source. Otherwise a change to the packing step does not invalidate the compressed output.

The last point is where the graph structure shows. If every step’s key is a hash of its inputs’ keys, the whole build becomes a Merkle tree: the key of the final output commits to the entire history of transformations that produced it. That property is what makes a cache correct under partial invalidation — a change anywhere in the chain changes every key downstream of it, and nothing else.

import hashlib, json

def step_key(tool, version, params, inputs):
    """Key for one build step: a digest over the tool identity,
    its parameters, and the keys (not bytes) of its inputs."""
    h = hashlib.sha256()
    h.update(tool.encode()); h.update(b"\0")
    h.update(version.encode()); h.update(b"\0")
    # Canonical form: sorted keys, no insignificant whitespace.
    h.update(json.dumps(params, sort_keys=True,
                        separators=(",", ":")).encode())
    h.update(b"\0")
    for name, key in sorted(inputs.items()):
        h.update(name.encode()); h.update(b"=")
        h.update(key.encode()); h.update(b";")
    return h.hexdigest()

Two details in that function carry most of its correctness. The separators between fields are not decoration: without a delimiter, a tool name ending in a digit and a version beginning with one can produce the same byte stream as a different pair, and two distinct inputs collide. And the parameter serialization must be canonical, with sorted keys and fixed separators, or two logically identical parameter sets hash differently and the cache misses forever.

Determinism, the prerequisite

A cache keyed on content is only useful if the same key produces the same output. If the tool is nondeterministic, embedding a timestamp, a temporary path, a random seed, or a dictionary iteration order into the output, then two builds with the same key produce different bytes, and the cache either returns a stale entry or misses on every run.

Asset tools are nondeterministic in predictable ways. An encoder that parallelizes block compression may assign blocks to threads in an order that changes which candidate wins a tie, producing different but equally valid output. A packer with a randomized starting layout produces a different atlas each run. A metadata writer may embed the source path, which differs between a developer’s machine and a build server.

The fixes are structural rather than clever. Tie-breaking must be defined by the data, not by arrival order. Randomization must be seeded from the input digest. Paths that appear in output must be relative or omitted. And the ordering of collections that reach the output must be stable — sorted by a content-derived key rather than by insertion order, since insertion order frequently depends on filesystem enumeration order, which differs between platforms.

Determinism is worth testing directly: build the same input twice into two directories and compare the bytes. A difference is either a nondeterminism to fix or an output that legitimately depends on something the key does not capture, and both are defects.

Invalidation that propagates

With a Merkle key per step, invalidation becomes mechanical. Change a source file, and its digest changes, so every step that consumed it gets a new key, and every step downstream of those gets a new key. Steps with no path to the change keep their keys and their cached outputs.

The cost of this scheme is that a change high in the graph invalidates a large subtree. Changing the project’s default compression setting invalidates every texture, because the setting is part of every step’s parameters. That is correct but expensive, and it is the reason pipelines separate parameters into ones that are genuinely global from ones that are per-asset: a global default that is copied into each asset’s parameters has the same effect on invalidation as changing every asset individually.

A more subtle case is a dependency that is not a file. A tool that reads a system font, an environment variable, or a network resource has an input the graph does not see, and a change to that input is invisible. The general response is to make implicit inputs explicit — hash the font, record the environment in the key — or to accept the risk knowingly for inputs that genuinely never change.

Cached outputs need a home, and the choice changes the economics. A local cache on the build machine is fast and useless across machines. A shared cache keyed by the same digest lets one machine’s work serve another’s, which is the property that makes the scheme pay for itself on a team. The shared cache must be content-addressed too, storing outputs under their key, so that a hit is a lookup rather than a comparison, and so that two different builds racing to populate the same key both write identical bytes.

Where the model breaks for assets specifically

General build systems assume inputs and outputs are files whose bytes are the whole story. Graphics assets violate that in three ways.

The tool’s own library versions matter and are not in the version string. A compressor that links a different version of a numerical library can produce different output from the same source with the same version reported. The rigorous response is to key on the tool binary’s own digest rather than on its reported version, which is what content-addressed build systems do for compilers. For asset tools distributed as large binaries, hashing the executable is cheap relative to the work it does.

Outputs are large and numerous. A texture mip chain plus its variants for three GPU formats is a set of files, and storing all of them in a cache multiplies storage by the variant count. Deduplication by content helps — the fallback variant and the compressed variant rarely share bytes — but the cache’s capacity becomes a real constraint, and eviction policy starts to matter. Evicting by last use is the usual choice; evicting by size biases against exactly the large textures that are most expensive to rebuild.

Partial outputs are legitimate. A texture build may produce level 0 quickly and the rest of the mip chain slowly, and a pipeline that can sample level 0 while the rest is in flight has a reason to cache partial results. A key-per-complete-output cache cannot express that, so streaming builds usually key per level rather than per texture, which turns each mip level into its own node in the graph.

Reading the failure modes

A cache that is wrong is worse than no cache, because the wrongness is silent. The failure signatures are worth knowing.

An output that is stale after a settings change means the settings were not in the key, or were in the key in a form that did not change: a boolean that flipped in a file the key hashes at a coarser granularity, or a default that the tool applies without it appearing in the recorded parameters.

An output that rebuilds on every run, with no input change, means nondeterminism in the tool or non-canonical serialization in the key. The two are distinguishable by building twice and comparing: identical outputs point at the key, differing outputs at the tool.

An output that is correct on one machine and wrong on another points at an implicit input such as an absolute path, an environment setting, a locale-dependent sort, or a floating-point mode. The last is the most insidious for graphics work, since a compiler that contracts multiply-adds differently can change an encoder’s block selection on borderline content.

And a cache that never hits across machines usually has an absolute path baked into the key. Making paths relative to a project root, and hashing relative paths rather than resolved ones, is the fix.

The structure of the problem is the same one that appears in every container format: a representation is only usable if the metadata that describes it is complete and travels with it. A compressed texture without its channel semantics, an atlas without its gutter width, a cache entry without the parameters that produced it — each is a payload whose interpretation depends on information that was left behind. The conditioning steps that produce these assets, from channel packing conventions through block-compressed texture bit rates, all end at the same requirement: record what the bytes mean, in the bytes, and the rest of the pipeline can be built on top.