Container Images as Frozen Environments and Their Limits

A container image solves the part of environment reconstruction that a lockfile cannot reach: it freezes the operating system packages, the system libraries, the compiler for anything built inside it, and the file layout, all under one name that resolves to a content digest. That is a large amount of coverage obtained cheaply, and it is the reason containers became the default transport for computational environments.

What the image does not freeze is equally definite, and it is worth being precise about because the failure modes are quiet. An image pins userspace. It does not pin the kernel, the hardware, the clock, the network, or the scheduler, and for numerical work each of those can change results.

Layers, digests, and what a tag actually names

An image is a manifest listing layer blobs and a configuration blob. Each blob is addressed by the digest of its bytes, typically SHA-256, written as sha256: followed by 64 hexadecimal characters. The manifest itself is addressed the same way, and that manifest digest is the image’s identity.

A tag such as myproject/analysis:latest is a mutable label in a registry that points at whatever manifest was pushed most recently. Pulling by tag is therefore not a reproducible operation: the bytes fetched today may differ from the bytes fetched at the time a result was produced, with no error and no warning. Pulling by digest is.

The layer structure is what makes this efficient and also what makes naive reasoning about it fail. Layers are stacked by an overlay filesystem, with later layers adding or replacing paths. The same set of packages installed in a different order, or with a different base image, yields different layer hashes even when the final filesystem is identical, because a layer records operations as a diff rather than as a state. So two images that behave identically can have different identities, and two images with a shared layer hash share those specific bytes but not necessarily the whole environment.

A consequence that matters for auditing: installing packages early in a Dockerfile and removing them later leaves them in the earlier layer. The file is gone from the final view; the bytes remain in the image and are retrievable. Layer ordering is thus both a size decision and a disclosure decision.

Building the same image twice

The digest identifies an image but says nothing about how it was made. Building the same Dockerfile twice can produce different bytes for reasons that have nothing to do with the source.

Package installation from a live distribution mirror is the largest source. apt-get install without version pins resolves against whatever the mirror currently holds; the mirror’s contents change as packages are updated and superseded. The resulting layer differs, and the environment differs in ways that may affect behaviour.

Timestamps are the second. Files written during a build carry modification times from the build, and any archive produced from them embeds those times. Build tools that set a fixed epoch for file timestamps remove this variability; the convention of a fixed timestamp for reproducible archives is old and widespread in distribution packaging.

Build ordering and parallelism are the third: a step that enumerates a directory, or a compiler that embeds an absolute path, or a linker that writes a build identifier derived from a temporary path, all produce nondeterministic bytes.

The practical response is to separate two claims that are often conflated. Bit-identical rebuild is achievable only with deliberate effort — pinned distribution snapshots, fixed timestamps, deterministic tooling — and is what the reproducible-builds project targets for software artifacts. Behavioural reconstruction is the weaker claim that a rebuild produces an environment in which the analysis computes the same numbers, and it tolerates different bytes. For scientific work, the second is usually the operative requirement, and a digest-addressed image satisfies it as long as the image is preserved rather than rebuilt.

Storing the digest rather than the Dockerfile is therefore the load-bearing practice. A Dockerfile is a recipe that may no longer produce the same image; a digest names bytes that a registry either has or does not.

The kernel is outside the image

A container is not a virtual machine. The process inside it runs on the host kernel, isolated by namespaces for filesystem mounts, process identifiers, network, and IPC, and constrained by cgroups for CPU and memory. The image contributes a filesystem; it does not contribute a kernel.

That has several consequences for numerical work.

The kernel version determines available system calls, and its configuration determines scheduler behaviour, memory overcommit policy, and transparent huge page handling. An analysis that depends on allocation patterns can behave differently across kernel versions with the same image. Transparent huge pages in particular change the fault behaviour and memory footprint of large arrays, and whether the feature is enabled or set to madvise is a host setting.

The CPU is passed through, not emulated, unless the container is run with emulation for a foreign architecture. Instruction set extensions available to the process are the host’s. Code compiled with runtime dispatch — the common pattern in numerical libraries, which select a kernel based on detected features — will take a different path on a host with AVX-512 than on one without it. Results computed through different vector kernels can differ in the last bits, and performance can differ by a large factor.

Thread counts and affinity come from the host and the cgroup limits. A library that inspects the machine’s core count at startup may see all host cores while being limited to fewer by the cgroup quota, and oversubscribe itself. Since thread count changes reduction order, it changes floating-point sums.

# Pull by digest and record it; the tag is not an identity
docker pull registry.example.org/analysis@sha256:3f9c...e21
docker image inspect --format '{{index .RepoDigests 0}}' \
    registry.example.org/analysis@sha256:3f9c...e21
# What the process actually sees, which the image does not decide
docker run --rm registry.example.org/analysis@sha256:3f9c...e21 \
    sh -c 'nproc; uname -r; grep -m1 "model name" /proc/cpuinfo'

The second command is the useful one: it prints the three facts that most often explain why two runs of the same image disagree.

Reproducibility of the run, not just the environment

Freezing the filesystem leaves execution nondeterministic in ways an analyst may not expect.

Time. The container inherits the host clock, including its time zone if /etc/localtime is mounted and its offset if the image sets one. Anything that timestamps output, seeds from time, or branches on a date behaves differently across days. Setting a fixed time zone in the image removes one class of variation; nothing removes the clock itself, so results that depend on it need their time recorded or their dependence removed.

Randomness. /dev/urandom draws from the host. A container does not isolate the entropy pool, so any code that seeds from the operating system receives different bytes each run. Seeding an explicit generator is the only fix, which is the point of Seeds, Generator State, and Reproducible Random Streams.

Hostname, user identity, and mount points. Absolute paths inside a container are fixed by the image’s layout, but bind mounts replace directories with host paths, and the mounted content’s ownership and permissions come from the host. A workflow that mounts a data directory at a path the code expects is relying on convention, and a mount with different permissions can fail in ways the image alone never would.

Resource limits. Memory and CPU quotas affect whether a computation completes, and some numerical libraries reduce their blocking or thread counts based on available memory. The image cannot record the limits, so a run that succeeded under one configuration may be killed under another with the same digest.

The network. Any step that fetches data, model weights, or a package at run time reintroduces exactly the variability the image was meant to remove. The mitigation is to fetch nothing during the run — bake data into a separate image layer or mount a content-addressed store — and to record input hashes as described in Content-Addressed Provenance Graphs for Derived Outputs.

Where containers are the wrong tool

Three situations argue against the container as the primary mechanism.

When the environment must match a scheduler-managed cluster, the cluster’s module system and MPI implementation are usually host-provided, and a container that ships its own MPI library will not interoperate with the launcher outside it. The container must be built against the host’s MPI ABI, which means the image is no longer portable across clusters.

When the deliverable is a small, auditable computation, a container is a large opaque artifact whose contents are hard to review. A lockfile plus a short script is more legible and just as reproducible for the package layer.

When the difference that matters is a system library version rather than a package version, a container helps only if the library is installed inside it rather than inherited through a device or a mount. GPU drivers are the standard case: the driver lives on the host and is exposed to the container, so the image pins the userspace library but not the driver, and a driver change can alter kernel selection and results.

The right claim for a container is narrow and useful: it reconstructs a userspace filesystem from identified bytes. Everything outside that filesystem — kernel, hardware, clock, entropy, limits — has to be recorded by other means, or its influence has to be removed from the computation. An image digest quoted in a methods section is a much stronger statement than a Dockerfile, and a much weaker one than “this result is reproducible” without qualification.