Passing Build State Between Jobs Without a Shared Workspace

A pipeline with several jobs needs the second job to see what the first produced. The familiar arrangement is a shared directory on a shared filesystem: the build job writes into it, the test job reads from it, and the whole thing works until the jobs land on different machines, or run concurrently, or one is retried and writes a second time. The workspace is a convenience that hides the actual transfer, and the hiding is what makes it fail in ways that are hard to attribute.

The transfer is over object storage, and it is not atomic by default

When jobs are scheduled independently, the producer’s output is uploaded to storage the orchestrator controls, and the consumer downloads it. That storage is an object store or a database, not a filesystem, and the semantics differ in ways that matter.

An upload is usually assembled from many requests: one per file, or one per chunk for large files. A job that fails or is cancelled partway leaves a partial artifact behind. Whether the partial object is visible to consumers depends on the store, and whether it is marked complete depends on the client. The safe pattern is to upload under a temporary name and rename or commit at the end, so consumers see either nothing or a complete object. Where the store does not offer an atomic rename, the equivalent is to write a small marker object last and have consumers require it.

The orchestrator adds a second layer. It records which artifacts belong to which job run, and it usually only publishes that record after the job exits successfully. The artifact may be physically present in storage while the consumer cannot address it, because the producer’s job has not been marked complete. That ordering is a feature; it prevents consumers from reading half-written state. It also means a job that produces an artifact and then fails in a later step loses the artifact, which is the intended behaviour for a build that fails and the wrong behaviour for a build whose artifact is usable and whose failure was in reporting.

Consumers should verify what they receive rather than assume. The cheapest verification is a size or checksum comparison against a value the producer recorded, which catches truncation, a partial upload accepted as complete, and a wrong-path download that fetched an empty object.

A digest manifest makes the consumer’s expectations explicit

The pattern that survives retries, partial uploads, and reordered jobs is a manifest: a small file, produced before the payload, listing each payload file with its digest and size. The producer uploads the payload, then the manifest, then the record that names the manifest. The consumer fetches the manifest, then the payload, then verifies.

import hashlib, json, pathlib

def write_manifest(out_dir: str, manifest_path: str) -> None:
    entries = {}
    root = pathlib.Path(out_dir)
    for p in sorted(root.rglob('*')):
        if not p.is_file() or p.is_symlink():
            continue
        h = hashlib.sha256()
        with p.open('rb') as f:
            for chunk in iter(lambda: f.read(1 << 20), b''):
                h.update(chunk)
        rel = p.relative_to(root).as_posix()
        entries[rel] = {'sha256': h.hexdigest(), 'size': p.stat().st_size}
    body = json.dumps({'version': 1, 'files': entries},
                      sort_keys=True, separators=(',', ':')).encode()
    # Written last, and uploaded last, so a reader never sees a manifest
    # describing files that are not yet present.
    pathlib.Path(manifest_path).write_bytes(body)

The manifest is also where the producer can state facts the consumer would otherwise have to assume: which target platform the artifact was built for, which compiler produced it, whether it is a debug or release build. Those are not derivable from the payload, and a consumer that assumes them will eventually assume wrongly. The manifest is small enough that storing it for every run costs nothing measurable, and it is the natural place to attach the provenance record discussed in Artifact Provenance from Commit to Deployed Digest.

Extraction is the dangerous step

A consumer that downloads an archive must extract it, and extraction from a producer the consumer does not fully trust is a known vulnerability class. The archive controls the entry names, and a name containing .. segments or an absolute path causes the extraction to write outside the intended directory. The tar implementations that unpack with a leading / stripped are protected against absolute paths and not against traversal, so a member named ../../etc/something still escapes. GNU tar refuses absolute names and, in recent versions, refuses members whose resolved path leaves the extraction directory unless explicitly permitted.

Symbolic links are a second vector. An archive containing a symlink to a directory outside the target, followed by a member whose path goes through that link, writes outside the target even though each individual member name looks safe. The defence is to extract into a fresh empty directory with symlink creation disabled or deferred, and to check each resolved path against the target root before writing.

Permissions are the third area. Tar records the mode in each header, and extraction applies it. A build artifact whose executables lost their execute bit is a common symptom of passing files through a copied directory rather than an archive, and it produces a consumer failure that looks like a missing binary. Archive formats preserve mode, which is a reason to pass an archive rather than a stream of files.

Format limits are worth knowing before choosing. The plain ustar header has a 100-byte path field and a 155-byte prefix field, so paths longer than about 255 bytes need the pax or GNU extension. Zip without the Zip64 extension cannot represent a file or archive at or above 4 GiB, and its per-entry timestamps have a two-second resolution with a floor in 1980. The two-second resolution means a zip used as a reproducibility artifact will differ between two builds that finish in different seconds unless timestamps are normalized, which is one reason tar with a fixed mtime is the more usual choice for build outputs.

What should travel, and what should be rebuilt

Passing everything is the default and the wrong one. Three categories are worth distinguishing.

Build outputs travel, because rebuilding them is the work the pipeline exists to avoid. These are the compiled binaries, the packaged distributions, the generated documentation.

Test reports travel, because the consumer of a report is a human or a dashboard, not a build step. Reports are small relative to the artifacts and their content is not an input to anything, so they belong in a separate artifact with its own retention rather than bundled with the payload.

The version control checkout does not travel. A job that needs the source reconstructs it from the commit, which is small, verifiable against the commit identity, and avoids shipping a working tree whose untracked files and local modifications are not what the pipeline intended to build. The same reasoning applies to dependency directories: they are reconstructible from the lockfile with the same cache key inputs that the build already computes.

The rule that follows is that an artifact contains things that cannot be cheaply recomputed, and everything else is recomputed. A test job that wants the checkout fetches the commit, not a copy of the tree from the build job.

Retries, duplicates, and the identity of an artifact within a run

A retried job produces a second set of artifacts. If the naming scheme is job-name alone, the retry overwrites or races with the first attempt, and a consumer that started downloading may get a mixture. Naming artifacts by the run identifier and attempt number makes the two attempts distinct, and pinning consumers to a specific attempt makes the mixture impossible.

Consumers should also be tolerant of a missing artifact in the specific case where the producer decided not to produce one. A build that emits artifacts conditionally, based on what changed, needs to express “nothing to pass” in a way the consumer distinguishes from “producer failed”. The manifest’s absence and an empty manifest are different states, and a consumer that treats them identically will either fail on legitimate no-change runs or proceed on genuine failures.

The deeper point is that a job boundary is a network boundary with a storage service in the middle, not a function call. Everything that crosses it can be missing, partial, duplicated, or stale, and a pipeline that treats the crossing as reliable is relying on a property the storage layer does not provide. The next problem in the same family is how a consumer skips the work entirely when someone else has already produced the same output, which is what a shared cache does and where Shared Build Cache Design begins.