Propagating Template Changes Across an Instantiated Fleet

The first tenants of a golden path take it as a snapshot. The template is rendered into their repository, committed, and from that moment the repository is theirs. Six months later the platform team changes the base image, tightens a security context, or replaces a deprecated API group, and discovers that the change has to be applied to a hundred independent repositories, some of which have been edited. The propagation model chosen at instantiation time determines whether that is a mechanical operation or a quarter of manual work.

Three models and what each one costs

The pull model renders a snapshot at instantiation and gives the tenant a copy. The tenant owns the copy entirely. There is no link back, no upgrade path, and no way for the platform to fix a defect in generated output except by asking tenants to re-render. Its virtue is that the platform can never break a tenant: a template change affects only future instantiations. Its cost is that every fix must be re-adopted by hand, and adoption is proportional to tenant count.

The push model keeps the template as a live dependency. The tenant’s repository holds only their parameter values, and the rendered output is produced at deploy time by a pipeline that resolves the current template version. A template change reaches every tenant at the next deploy. Its virtue is that fixes propagate without tenant action. Its cost is that a template defect becomes a fleet-wide defect at the next deploy, and the platform can break every tenant at once.

The hybrid model pins a version per tenant and lets the tenant choose when to move. The tenant’s repository records the template version, the pipeline renders that version, and the platform publishes new versions that tenants adopt on their own schedule. This is the model most platforms converge on, because it separates the two decisions that the other models conflate: who bears the cost of adopting a change, and who bears the cost of a change being wrong.

The hybrid’s own cost is that the platform must support several versions simultaneously. If tenants pin at their own pace and the platform supports every version ever published, the support surface grows without bound. The usual resolution is a support window: the current major version plus the previous one, with an announced date after which the oldest is retired. The window is the platform’s leverage, and it only works if the retirement is enforced, which means eventually refusing to deploy a retired version rather than warning about it indefinitely.

What a version number must distinguish

A template version has to communicate more than “newer”. The distinctions that matter to a tenant are whether their parameters remain valid, whether the rendered output changes in a way that affects behaviour, and whether the change is reversible.

A version that only adds a new optional parameter with a default that reproduces existing output is safe for every tenant and requires no action. A version that changes a default value, adds a required parameter, or removes a parameter is not. Encoding the distinction in the version number itself lets the pipeline decide whether to warn, block, or auto-adopt.

A workable convention is that the major number changes when the parameter schema changes incompatibly, the minor number changes when rendered output changes in a behaviourally significant way, and the patch number changes when the change is confined to generated content that has no runtime effect, such as a comment or an annotation used only for provenance. The version number alone does not tell a tenant whether their particular parameters still produce valid output, so the pipeline must also evaluate the schema against the tenant’s recorded parameters and report the specific violations before rendering.

That evaluation is the step most often skipped. The template’s schema validates at instantiation, and the version bump revalidates at upgrade, which produces a precise error such as “parameter ingressClass is no longer accepted; the replacement is gatewayRef”. Without it, the tenant discovers the incompatibility as a render failure at deploy time, with an error that names the renderer’s internals rather than the parameter.

Three-way merge and why drift decides the strategy

A tenant repository that has never been edited can be upgraded by re-rendering and replacing. A repository that has been edited requires a merge, and the merge is three-way: the old rendered output, the new rendered output, and the tenant’s current file.

The merge base is the old rendered output, which is why the provenance stamp matters. Without a stored copy of what the platform generated at the pinned version, the merge has no base and degenerates into a two-way comparison between the new output and the tenant’s file, where every tenant edit looks like a conflict.

With a base, the merge is mechanical and the conflicts are informative. A line the tenant changed and the template did not change is kept. A line the template changed and the tenant did not is taken. A line both changed is a conflict that a human resolves, and the resolution should be recorded so the next upgrade does not ask again.

The practical problem is that tenants edit generated YAML constantly, and YAML merges badly: reordering a list of environment variables is a semantic no-op and a textual conflict. Two mitigations reduce the volume. Generated files can be marked with a header that discourages editing and a pipeline check that reports drift, so the platform knows which repositories will conflict before starting the rollout. And generated output can be structured so that the parts tenants are expected to change are separate files from the parts the platform owns, which turns most upgrades into a replacement of the owned files and no merge at all.

Sequencing a breaking change

The safest shape for a breaking template change is not a single release. It is a sequence in which each step is individually safe for every tenant.

The first step introduces the new parameter alongside the old one and makes both work. The rendered output uses the new value if present and falls back to the old. Tenants who do nothing are unaffected.

The second step makes the pipeline warn when the old parameter is used, naming the replacement and the retirement date. This is where adoption actually happens, because the warning appears in the tenant’s own pipeline output rather than in a platform announcement.

The third step, after the window closes, makes the old parameter an error at render time. The error names the replacement and the migration, and it fires before deployment, so no tenant discovers the change during a rollout.

The fourth step removes the old parameter from the schema and from the renderer.

Each step is reversible until the fourth. A defect discovered at step three can be walked back by relaxing the error, which restores the previous behaviour without requiring tenants to re-edit. This is the property that makes the sequence worth its length: it converts a breaking change into four non-breaking ones.

The failure mode of the sequence is abandonment after step two. Warnings are cheap to add and easy to leave in place, and a parameter that has been deprecated for two years without becoming an error has taught tenants that deprecation warnings are noise. The retirement date in the warning is only credible if the date is enforced, which means the platform must track which tenants still use the old parameter and contact them, not merely emit the warning into logs nobody reads.

Verifying the fleet rather than the template

Template tests verify that the template renders correctly. They do not verify that the fleet is in the state the template version implies. The gap matters because the fleet is what the platform is judged on.

The verification that closes it is a periodic scan that reads each tenant’s pinned version, the version they would get if they upgraded today, and the drift between their repository and their pinned render. Three numbers per tenant: pinned version, available version, drift status. The first identifies who is on a retired version, the second identifies who is behind, and the third identifies who will conflict when they move. A rollout plan can then be sequenced by conflict risk rather than by tenant name.

This scan also detects the tenant who left the path entirely: their pinned version is old, their drift is total, and their workloads no longer resemble the template’s output. That tenant is not a failure of the template; they are a workload the platform supports as an exception, and the platform should know how many of those exist. The mechanism that keeps exceptions from silently multiplying is the versioned contract of the golden path, and the point at which a path is retired rather than maintained is covered in sunsetting a golden path.