Placement Constraints and the Anatomy of an Unschedulable Workload

A workload that stays pending produces one event, and the event is a list: every node, and the reason each one rejected it. Reading that list as a set of independent failures is the natural mistake, and it is wrong, because most of the constraints that appear in it are not local properties of a node. They are properties of the placement of other workloads, and they interact. A pod can be admissible on every node in isolation and admissible on no node simultaneously.

The constraints divide into three families with different semantics, and getting a placement right requires knowing which family a given rule belongs to.

Taints, tolerations, and the direction of repulsion

A taint is a property of a node that repels workloads. It has a key, a value, and an effect, and the effect is the interesting part: NoSchedule means new workloads without a matching toleration are not placed there, PreferNoSchedule means the scheduler scores the node lower but will use it, and NoExecute means workloads already running there without a toleration are evicted, in addition to blocking new ones.

A toleration is a property of a workload that matches a taint. Tolerations do not attract; they merely neutralise. A pod that tolerates every taint in the cluster is not preferentially placed on tainted nodes, it is merely not repelled from them. This asymmetry is deliberate, and it means that dedicating a set of nodes to a class of workload requires two rules: a NoSchedule taint on the nodes, and both a toleration and a node affinity on the workloads that belong there. The taint keeps everyone else out; the affinity keeps the intended workloads in. With only the taint, the intended workloads are free to run anywhere and will.

NoExecute is the only effect with a time dimension. A toleration can specify tolerationSeconds, and a pod that tolerates a NoExecute taint with a bounded duration stays for that long after the taint is applied and is then evicted. This is the mechanism behind graceful handling of node problems: the node controller taints an unreachable node NoExecute, pods with a bounded toleration keep running for the grace window in case the node comes back, and pods without one are removed immediately.

The pairing of taints with eviction is also the reason a cordoned node behaves differently from a deleted one. Cordoning applies an unschedulable flag that blocks new placements while leaving existing workloads untouched; draining adds eviction of the existing workloads. The distinction is what makes a rolling node replacement possible at all: the node stays functional long enough to move work off it.

Node affinity, pod affinity, and the two directions of attraction

Node affinity constrains a workload to nodes by label. It comes in required and preferred forms, and the required form is a hard filter while the preferred form is a score. The operators are the usual set-membership ones: In, NotIn, Exists, DoesNotExist, plus the numeric Gt and Lt. A required node affinity that matches nothing is an unsatisfiable filter, and every node appears in the event with the same reason.

Pod affinity and anti-affinity are the same operators over a different domain: instead of matching a node’s labels, they match the labels of pods already placed in a topology domain. A topology key names the node label that defines the domain: kubernetes.io/hostname for one node, or a zone label for a whole zone. The rule then reads “there must exist (or must not exist) a pod matching this selector in the domain containing this candidate node”.

Three consequences follow, and all three are operational.

First, required pod anti-affinity with a hostname topology key is a hard cap on replicas per node of one. A workload with required anti-affinity against its own selector can have at most one pod per node, so a replica count above the node count leaves pods permanently pending regardless of capacity. The event will show resource fit succeeding on many nodes and anti-affinity failing on all of them, which is the signature of this configuration.

Second, pod affinity’s filter is not local, so its cost is not local. Counting matching pods per topology domain for each candidate node is the expensive part of the filter pass, and it is why schedulers maintain per-domain selector counts rather than walking the pod list per node.

Third, pod affinity is evaluated against pods that are already scheduled, so a set of workloads that require each other forms a deadlock: each waits for the other to be placed first. The scheduler has no mechanism to break this, because placing one would violate its constraint. The standard resolution is to make one side preferred rather than required, or to introduce an ordering so that a workload with no affinity requirement is placed first and becomes the anchor.

Topology spread and the constraint that is not a filter

Topology spread constraints express a distribution requirement rather than a presence requirement: for a given topology key and a maximum skew, the difference between the number of matching pods in the most-populated domain and the least-populated domain must not exceed the skew. The constraint is evaluated with the candidate node’s domain treated as receiving one more pod.

Two properties make this family different from the others. It is inherently global, since a decision on one node changes the skew for every other node, and it is unsatisfiable in a way that is easy to misread. A spread constraint with maxSkew: 1 over three zones and a replica count of six permits two per zone; a replica count of seven does not, because two-two-three has a skew of one and is fine, but the seventh pod cannot be added to any zone without making it three against one, which is a skew of two. The correct reading is that the constraint does not fail because of node capacity; it fails because the desired distribution is arithmetically impossible at that replica count.

The whenUnsatisfiable field decides whether the constraint filters or merely scores. DoNotSchedule makes it a hard filter, and it is the setting that produces the pending-pod behaviour above. ScheduleAnyway turns it into a score, which means the workload is always placeable and the distribution is best-effort. The second is almost always what a deployment wants, because a hard spread constraint converts a scaling event into an outage: the deployment raises replicas, the new pod cannot satisfy the skew, and it stays pending while the old pods keep serving.

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: topology.kubernetes.io/zone
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: web

A further subtlety is the interaction between the constraint’s minDomains setting and the number of domains that actually exist. With minDomains set, the skew calculation treats missing domains as having zero matching pods, so a cluster that has lost a zone is penalised for it. Without it, the calculation only considers domains that exist, so a two-zone cluster satisfies a spread constraint that was written for three. Neither is wrong; they encode different intents, and the intent determines whether a degraded cluster should keep accepting work.

Preemption: the only way out of a hard filter

When no node passes the filters, the scheduler can consider preemption if the workload has a priority higher than some running workloads. The procedure is: for each node, find a minimal set of lower-priority victims whose removal would make the node feasible; choose the node whose victims have the lowest priority; evict them; and leave the preempting workload pending until the victims have actually terminated and the node’s accounting reflects it.

The word “minimal” carries the cost. Selecting the smallest victim set is a combinatorial problem, and implementations approximate it by considering candidates in order of increasing priority and stopping at the first set that works. The approximation can pick a set that is not minimal, and it can fail to find a set that exists, in which case the workload is reported unschedulable despite a feasible preemption existing.

Preemption does not respect the constraints of the victims. A preempted pod is deleted; it is then re-scheduled like any other pod, and if its own constraints cannot be satisfied elsewhere it stays pending, having lost its place. A cluster that relies on preemption for priority inversion therefore needs to check that the low-priority workloads are actually movable. A common failure is a set of system-critical pods with anti-affinity rules that make them unplaceable once evicted, so preemption converts a scheduling problem into a partial outage.

Preemption also respects the disruption budget of the victims’ controllers only partially. The scheduler evicts; whether the eviction is graceful depends on the pod’s termination grace period and on whether the controller has a budget that would block it. A pod protected by a budget that permits no disruption is still evicted by preemption, which is one of the few places where a budget does not apply.

Reading the event as a diagnostic

The pending event lists, per node, the filter that rejected it. The useful reading groups the failures rather than counting them. If every node reports the same resource failure, the request is too large for the cluster, and the answer is capacity or a smaller request. If nodes report a mix of resource successes and affinity failures, the constraint is the binding one. If every node reports a different topology reason, the spread constraint is arithmetically unsatisfiable. If all nodes report the same taint, a toleration is missing.

The scheduler’s own view of the failure is coarse: it reports the filters that ran, not why the configuration was written that way. Distinguishing “no node has capacity” from “the configuration cannot be satisfied by any placement” is the operator’s job, and the two have opposite remedies. The first is answered by the capacity accounting in resource requests, limits, and scheduler accounting; the second is a configuration defect that more capacity will never fix.

One class of failure sits outside both: the workload is feasible, a node is available, and the pod is still not bound. That is a failure of the node itself, or of the scheduler’s view of it, and it is the province of heartbeats, leases, and node failure detection.