Sync Ordering and Pruning in a Git-Driven Control Loop

Storing desired state in Git turns deployment into a comparison: render the repository, read the cluster, and make them agree. The comparison is the easy part. Two decisions around it are not. The first is order, because a set of manifests is not a set of independent objects, and applying them in the wrong sequence produces failures that resolve themselves on the next pass or, worse, failures that do not. The second is deletion, because “make them agree” includes removing objects that exist in the cluster and no longer exist in the repository, and a deletion driven by a text diff can remove far more than the diff suggests.

A reconciler that gets both wrong still converges, eventually, in the common case. The failures appear at the boundaries: first install, disaster recovery, and the day someone renames a directory.

Synced and healthy are different claims

Every implementation of this pattern reports two states, and conflating them is the source of most operational confusion.

Synced means the manifests in the repository have been applied to the cluster. The API server accepted every object; the desired spec is now what Git says.

Healthy means the objects are working. A deployment whose pods are in ImagePullBackOff is synced and unhealthy. A custom resource whose controller has not yet created the underlying cloud resource is synced and unhealthy. A job that has not completed is synced and unhealthy.

The distinction matters because the ordering mechanism depends on it. If waves advanced on “synced”, the next wave would start before the previous one’s controllers existed, and every dependent resource would be rejected or ignored. Waves advance on health, which means the health assessment is part of the ordering contract and has to know what healthy means for each kind. An assessment that reports healthy for any object it does not understand turns the ordering guarantee into a timing coincidence.

Health assessment is also where the loop is most likely to hang. A resource whose controller never updates its status is never healthy, so the sync never advances, and the failure looks like a stalled deployment rather than a missing controller.

Why kind ordering only covers half the dependencies

Reconcilers ship with a built-in order over kinds: namespaces before namespaced objects, custom resource definitions before the custom resources that use them, and cluster-scoped configuration before the workloads that depend on it. That order is derived from the API’s own structure and it is correct as far as it goes.

It does not cover dependencies that are not structural. A deployment that mounts a ConfigMap needs the ConfigMap to exist, and both are namespaced core objects with no ordering relationship in the kind order. A validating webhook that rejects pods needs its own backend running before any pod is created, which is a dependency in the opposite direction from the one the kind order encodes and can produce a deadlock: the webhook’s deployment cannot start because the webhook is not ready, and the webhook is not ready because its deployment is not running.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: workloads
  namespace: platform
spec:
  dependsOn:
    - name: infrastructure
  wait: true
  prune: true
  healthChecks:
    - apiVersion: apps/v1
      kind: Deployment
      name: policy-agent
      namespace: platform

The mechanisms for expressing non-structural order are explicit dependencies between groups of manifests, or numbered waves within a single group. Both are annotations on the desired state rather than inferences from it, which means they are maintained by hand and drift when the manifests are restructured. A wave number that was correct when a chart had three resources is often wrong after it has thirty.

The alternative to explicit ordering is to apply everything and let the loop retry. Because the reconciler is level-triggered, a rejected custom resource will be retried on the next pass and will succeed once the CRD exists, so convergence does happen. What it costs is a period during which the cluster contains a partial version of the change, error events that mask real failures, and no way to distinguish “not yet” from “never”. For a first install into an empty cluster that cost is acceptable; for a change to a running production set it is not, because the partial state is live traffic.

Pruning is a deletion API with a text front end

Pruning compares the set of objects the repository claims to own against the set the cluster contains, and deletes the difference. The comparison is only safe if “the set the repository claims to own” is well defined, and that is an ownership problem before it is a diff problem.

Ownership is recorded in labels and annotations that the reconciler writes when it creates or adopts an object. Without them, the reconciler cannot distinguish an object it created from one a human created in the same namespace, and pruning becomes a delete-everything-that-is-not-in-Git operation. With them, pruning is bounded by the recorded ownership, and an object created by hand survives because it was never claimed.

The label-based approach has a failure mode that is easy to trigger: two reconcilers managing overlapping sets. A second application that also claims objects in the namespace will see the first application’s objects as unowned by it, and either ignore them or, depending on configuration, prune them. The defence is a namespace-per-application boundary or an explicit ownership hierarchy, and it has to be decided before the second application exists.

Prune order is the reverse of apply order: objects in later waves are deleted first, so dependents go before their dependencies. The case that makes this matter is the custom resource definition. Deleting a CRD deletes every custom resource of that type, through the API server’s own garbage collection, regardless of whether those resources are in Git. A prune that removes a CRD because a directory was renamed therefore removes data that no diff mentioned. Most reconcilers offer a per-resource opt-out from pruning, and CRDs are the resource type for which it should be the default rather than an exception.

Finalizers interact with pruning in the same way they interact with any deletion: an object with a finalizer whose controller is gone stays in a terminating state forever, and a sync that waits for pruning to complete waits forever. The reconciler’s own finalizer, which ensures cluster resources are removed before the application record is, is a common source of stuck deletions when the application is deleted while its controller is unhealthy.

The reconciler and the human disagree, silently

When a human edits a live object, the next sync reverts the edit. That is the design: Git is the source of truth, and the loop enforces it. The operational problem is that the revert is invisible unless someone is watching events. A debugging session that changes a replica count to test a hypothesis finds the change undone within a sync interval, and the hypothesis is never tested.

Automatic sync makes this immediate; manual sync makes it a pending diff that someone must approve. Neither is wrong, and the choice determines whether the platform is safe against accidental edits or responsive to deliberate ones. What matters is that the interval is known. A sync interval of minutes and an interval of hours produce very different debugging experiences, and the interval is a property of the platform’s configuration that tenants should be able to look up rather than infer.

The field-level version of the same conflict is server-side apply. Each field has an owner, and a write that changes a field owned by another manager is a conflict unless forced. This is stricter than the whole-object replacement of client-side apply, and it surfaces precisely the disagreements that the older approach silently resolved. Adopting it requires deciding which manager owns which fields, which in a GitOps setup means deciding which parts of an object the repository controls and which parts a controller or a human controls. The answer is usually that the repository owns the spec and the controller owns the status, and that any field outside that split needs an explicit owner.

Where the loop’s guarantees stop

A Git-driven loop guarantees that the cluster converges to the repository, provided the repository is readable, the reconciler is running, and the objects it produces are valid. Each of those is a real dependency.

If the repository is unavailable, the cluster keeps running its last applied state, which is the intended behaviour and also means that an outage of the Git host is an outage of the ability to change anything. If the reconciler is down, the same is true with less visibility. If the manifests are invalid, the sync fails and the previous state persists, which is a safe failure but a silent one unless the failure is surfaced.

The guarantee that is genuinely absent is atomicity. A sync is a sequence of API writes with no transaction around them, so a failure in the middle leaves a mixture of old and new objects. The reconciler will retry, and the retry will converge, but the interval contains a state that no commit describes. Platforms that need the interval to be invisible put the change behind a switch that the workloads read, so the manifests can be applied in any order while the behaviour changes at one point. That is a property of the desired state, not of the sync, which is why it has to be designed into the manifests rather than delegated to the ordering mechanism.