Ephemeral Preview Environments: Naming, Lifetime, and Cost Control

A preview environment is a full instance of an application, created for one change, reachable by a URL, and destroyed when the change is done. It compresses the whole platform into a shorter lifetime: the same template, the same admission rules, the same quotas, but with a lifetime measured in hours rather than months. That compression exposes the decisions that a long-lived environment can hide, because anything that leaks is visible within a day.

Deriving a name that survives a retry

The first decision is the name, and it has more constraints than it appears to. It must be unique across concurrently open changes, stable across retries of the same change, and valid as a DNS label, because it becomes part of a hostname.

Uniqueness and stability are in tension. A name derived from a random value on each attempt is unique and unstable: a retried pipeline creates a second environment, and the first one leaks. A name derived from the branch name alone is stable and not unique, because two changes can share a branch name across forks, and because a branch name may exceed the length limit or contain characters that are legal in git and illegal in DNS.

The workable derivation is a deterministic function of facts that identify the change: a repository identifier, the change number, and a hash of the source reference truncated to a fixed length. All three are known before the environment is created, none of them changes when the pipeline retries, and the hash absorbs the length and character-set problems of the source reference. Truncation to a bounded length is not cosmetic; a DNS label is limited to 63 octets, and a name that exceeds it is rejected by the API server with an error that mentions the field rather than the length.

The label has to be lowercase alphanumerics and hyphens, must start and end with an alphanumeric, and cannot contain consecutive hyphens under the hostname rules in RFC 1123. A branch called feature/API_v2 becomes feature-api-v2 after normalization, and the normalization must be applied before hashing so that two spellings of the same branch do not produce two environments.

Lifetime bound to an observable fact

The second decision is what ends the environment’s life, and the choice determines whether cleanup is reliable.

A lifetime bound to a timer is simple and wrong in both directions. An environment created for a change that is merged in an hour lives for the rest of its time-to-live, wasting capacity. An environment for a change under review for a week expires mid-review, and the reviewer loses the instance they were testing.

A lifetime bound to an event is better, and the event must be one the platform can observe rather than one it is told about. The merge or close of the change is observable through the source control system’s API. The deletion of the branch is observable. Both are facts the platform can poll or subscribe to, and both are authoritative in a way that a pipeline’s final step is not: a pipeline that is cancelled, or that fails before its cleanup step, never reports that the change is done.

The mechanism is a controller that lists environments, reads the change state for each, and deletes the ones whose change is merged or closed. This is the same reconciliation pattern described in level-triggered reconciliation, applied to a resource whose desired state is “absent”. The controller’s pass frequency bounds the leak window, so a pass every few minutes is enough for cost control and short enough that a reviewer rarely encounters a stale instance.

A secondary bound is still needed for the case where the change is never merged or closed: an abandoned change, a change in a repository that was archived, a change whose author left. An inactivity timeout on top of the event binding covers it, where inactivity is measured by the environment’s own request log rather than by the change’s comment activity.

Cost, and why the environment must be a real instance

There is a temptation to make preview environments cheap by making them partial: one replica instead of three, no database, mocked dependencies. Each reduction lowers cost and lowers fidelity, and fidelity is the only reason the environment exists. An environment that cannot reproduce a bug caused by interaction with a real dependency has no diagnostic value.

The workable compromise is a scaled-down but structurally complete instance: one replica per service, a real database at reduced capacity, real dependencies pointed at their staging instances. The scaling is a parameter of the template, not a different template, so the preview differs from production in size and not in shape. This matters because a preview environment that uses a different template will diverge from production in ways the preview cannot detect.

The cost ceiling is arithmetic. An environment of $s$ services, each with a request of $r$ cores and $m$ bytes, on a pool with $N$ nodes of capacity $c$ and $M$, supports at most

$$\min\left(\left\lfloor \frac{N \cdot c}{s \cdot r} \right\rfloor, \left\lfloor \frac{N \cdot M}{s \cdot m} \right\rfloor\right)$$

concurrent environments, before any headroom. With headroom for the platform’s own workloads, the real number is lower. A platform that allows unlimited concurrent environments will discover this ceiling as a scheduling failure at the least convenient moment, which is when several changes are open at once.

Two policies bound it deliberately. A per-tenant concurrency limit, so one team cannot occupy the whole pool with a batch of open changes. And a queue with a bounded wait, so the eleventh environment waits rather than failing, and the tenant knows it is waiting. The queue is preferable to the hard failure because the request is legitimate; it is only the timing that is wrong.

The cleanup that ownership gives for free

Deleting the top-level object is easy. Deleting everything the environment created is the part that leaks, and the mechanism that makes it reliable is ownership: every object created for the environment is created with an owner reference to a single root object, and the platform’s garbage collector deletes the dependents when the root is deleted.

This requires discipline at creation time rather than at deletion time. Any object created without an owner reference is orphaned by deletion and must be found by a sweep, and sweeps are unreliable because they depend on labels being applied consistently. An object created by a controller that does not propagate the owner reference, or created through an API path that strips it, is invisible to the collector and visible to the billing report.

The practical pattern is a single namespace per environment. The namespace is the root object, deleting it deletes its contents, and the collector’s work reduces to deleting namespaces. This also gives the environment its own quota, its own network policy, and its own role bindings, which is exactly the isolation unit described in tenant isolation in a shared platform. The cost is that namespaces are not free: each one carries a set of default objects and a share of the control plane’s watch traffic, and a platform with thousands of concurrent namespaces pays for them in API server memory and in the latency of every list operation.

What the environment must report

An ephemeral environment is only useful if the people who need it can find it. The URL has to be published where the reviewer is looking, which is the change itself, and it has to be stable enough that a link posted in a comment still resolves after the environment is recreated.

Two pieces make that work. A status write back to the source control system, posted when the environment becomes ready and updated when it is recreated, so the URL in the change reflects the current instance rather than a stale one. And a readiness condition on the environment object, so a pipeline that waits for the environment waits for a fact rather than for a fixed delay.

The failure this prevents is the one where the environment is ready but nobody knows, so the reviewer tests against a half-started instance and reports a defect that does not exist. A condition with a reason and a timestamp is the difference between a preview that is trusted and one that is used once and abandoned.

The sizing arithmetic that decides how many environments a pool can hold is the same as the quota arithmetic for shared pools. The question of whether these environments belong in the same cluster as production is taken up in blast radius and cluster segmentation.