Owner References, Finalizers, and Why Deletion Is a Protocol

Deleting a deployment is a single API call and takes effect immediately in the sense that the object is gone from every subsequent read. The pods it created are not gone, and cannot be, because deleting a container takes time: a graceful shutdown, a volume detach, a network rule removal. Each of those steps can fail or hang. An orchestrator that treated deletion as a state transition rather than as a process would have to choose between abandoning resources and blocking the caller, and both are unacceptable. What it does instead is make deletion a two-phase protocol with an explicit intermediate state, and use a separate controller to drive the second phase.

Identity is a name plus a generation, and both are needed

Every object has a name that is unique within its kind and namespace, and a identifier that is unique across the cluster’s lifetime. The identifier is assigned at creation and never reused. Names, by contrast, are reusable: a deployment deleted and recreated with the same name is a different object with the same name.

Almost every ownership bug comes from confusing the two. A controller that remembers “the pod named web-abc” and later acts on a pod with that name may be acting on a different object, because the original was deleted and a replacement was created with the same name by a different controller. A controller that remembers the identifier cannot make that mistake.

This is why every ownership link is expressed as a reference containing both the name and the identifier. The reference is meaningful only if both match: the object named web-abc with identifier X is the dependent of its owner, and an object named web-abc with identifier Y is not. The comparison is a single field check at admission, and it is what makes it safe to recreate objects with reused names.

metadata:
  name: web-6d4b-x2l9p
  finalizers:
    - example.com/release-volume
  ownerReferences:
    - apiVersion: apps/v1
      kind: ReplicaSet
      name: web-6d4b
      uid: 4c1d3f8a-0b2e-4a6f-9d5e-77c0a1b2c3d4 # identity, not just the name
      controller: true
      blockOwnerDeletion: true

Owner references and the direction of the link

The dependent holds the link, not the owner. A pod carries a list of owner references, each naming a kind, a name, and an identifier; the deployment does not carry a list of its pods. The direction is chosen for two reasons.

The first is cardinality. A deployment may own thousands of pods, and a list on the owner would be a large, hot, frequently rewritten field. Every pod creation would be a write to the deployment, serialised against every other pod creation. A reference on the dependent is one small field written once, at the dependent’s creation, and never modified. The cost moves to the read side: finding an owner’s dependents requires an index over the reference field, maintained by the API server.

The second is authority. An owner reference can only be set at creation or by a controller with permission on both objects, and the API server validates that the named owner exists and that the namespace rules are respected. A dependent cannot claim an owner arbitrarily, and a cross-namespace owner reference is rejected outright, which prevents a namespaced object from being owned by something in another namespace whose deletion would silently cascade across the boundary.

The reference also carries a flag that decides what happens on the owner’s deletion. With the default, deletion of the owner cascades: the dependents are deleted too. With the orphan flag, the dependents survive and their owner references are stripped, which is the mechanism behind keeping a workload running while its managing object is replaced.

Cascading deletion itself has two modes, and the distinction is the difference between a blocking and a non-blocking call. Foreground cascading marks the owner with a deletion timestamp and a finalizer, then deletes the dependents, and removes the owner only after all dependents are gone, so the owner remains visible, in a terminating state, until its children are gone. Background cascading deletes the owner immediately and lets the garbage collector find and delete the dependents afterwards, which means there is a window in which the dependents exist with no owner. A caller that waits for the owner’s disappearance and then assumes the dependents are gone is wrong in the background case and right in the foreground case.

The garbage collector as a reconciliation loop

The collector is an ordinary controller with the same structure as any other: it watches objects, maintains an index of owner references, and reconciles. It does not run on a timer that sweeps the cluster; it reacts to events and periodically resyncs, and its work is to compare each object’s owner references against the existence of the named owners.

For a dependent whose owner no longer exists, the collector’s action depends on the deletion mode recorded at the time. If the object was meant to be cascaded, the collector deletes it. If it was meant to be orphaned, the collector removes the dangling reference and leaves the object. The distinction is recorded in the owner reference itself as a controller flag, so the collector can decide without consulting a history.

The collector’s index is the reason owner lookups are cheap, and it is also the reason a collector that restarts has a startup cost proportional to the number of objects. On startup it must list all objects of the kinds it manages and build the index before it can act, for the same reason any controller must: before the initial list completes, every owner appears absent and every dependent looks orphaned. A collector that acted during that window would delete the entire cluster’s dependents, which is why the initial-sync gate described in the reconciliation loop is not an optimisation here but a correctness requirement.

Finalizers: the veto on completion

A finalizer is a string in the object’s metadata that blocks the object’s removal. While at least one finalizer is present, a delete request sets the object’s deletion timestamp and returns success, but the object is not removed. It remains readable, it still appears in lists, and it still counts against any quota it was charged against. When the last finalizer is removed, the object disappears.

The deletion timestamp is the signal. A controller that owns a finalizer watches for objects with a non-null deletion timestamp and performs its cleanup, then removes its finalizer. The pattern makes the cleanup idempotent by construction: the cleanup may be attempted repeatedly, and it must be safe to do so, because the only acknowledgement is the removal of the finalizer and the controller cannot know whether a previous attempt succeeded.

The set of finalizers is a set of claims, and each is owned by a different controller. This is what allows several independent cleanups to be sequenced without any of them knowing about the others: the load balancer controller removes its finalizer when the backend is deregistered, the volume controller removes its when the attachment is released, and the object disappears when the last one is done. The ordering is not specified, which is deliberate: a cleanup that requires another cleanup to have happened first cannot be expressed this way and must instead be sequenced by one controller holding both responsibilities.

The failure mode is obvious and common. A finalizer whose controller is not running, or whose cleanup cannot succeed, blocks deletion forever. The object stays in the terminating state, its name is not released, and any object that depends on it, whether a namespace containing it or an owner whose foreground deletion waits for it, is also blocked. The cascade is the dangerous part: a namespace cannot be deleted while any object in it has a finalizer, so one stuck custom resource makes an entire namespace undeletable, and the namespace’s own finalizer keeps it visible. Recovering requires either restoring the controller or removing the finalizer by hand, and the latter skips the cleanup the finalizer existed to perform.

The namespace as a deletion unit

Deleting a namespace is not a bulk delete of its contents. The namespace object gets a deletion timestamp and a finalizer; the namespace controller then lists the namespace’s contents and deletes them, in an order that respects their own finalizers, and removes its own finalizer only when the namespace is empty.

Two consequences follow. The first is that namespace deletion is eventually consistent and can be slow: a namespace with many objects takes time proportional to the number of objects and to the slowest finalizer among them. The second is that the namespace is not a transaction. Objects created in the namespace after the deletion began are also deleted, because the controller’s list is repeated, but an object created by a controller that is still running may race the deletion, producing a stream of creations and deletions until the controller notices the namespace is terminating and stops.

The API server refuses new creations in a namespace with a deletion timestamp, which closes most of the race, but not the one where an object is created between the namespace’s admission check and the deletion’s propagation. The window is small and the outcome is a deleted object, so the practical consequence is churn rather than an orphan.

What ownership does not express

Owner references express a containment relation, not a dependency graph. Two objects can depend on each other’s existence without either owning the other, and nothing in the mechanism detects the cycle or orders the deletions. A service and the workload it selects are the standard example: neither owns the other, deleting either leaves the other functional but useless, and the collector does not care.

The mechanism also cannot express ownership by something outside the cluster. A volume backed by an external storage system is owned by the object that claims it, and deleting that object releases the claim; whether the backing store is then deleted depends on a reclaim policy that lives in a different object and is not part of the ownership relation. The result is that deleting an object can leave resources behind that no controller in the cluster can see, which is a leak by design rather than by accident. The alternative, deleting external data on object deletion, is worse.

The most consequential limitation is that ownership is not a lock. Two controllers can both believe they own a set of objects, and the mechanism will not arbitrate; whichever acts last wins, and the objects oscillate. Preventing that requires that only one controller ever creates objects of a given kind for a given owner, which is a property of the controllers’ design rather than of the API. How the control plane arranges for exactly one instance of a controller to be active is the subject of leader election and control-plane quorum.