Cluster Segmentation: Choosing a Blast Radius Instead of Accepting One

A cluster is a failure domain whether or not anyone designed it as one. The API server, the controller manager, the scheduler, and the datastore behind them are shared by every workload in the cluster, so a fault in any of them is a fault for all of them. Deciding how many clusters to run is therefore a decision about which failures are allowed to be simultaneous, and the answer depends on how the failure classes actually propagate.

Three failure classes that behave differently

Node failure is local. A node’s kubelet stops reporting, the control plane marks the node unreachable, and after a grace period the pods on it are marked for deletion and rescheduled elsewhere. The blast radius is bounded by the number of pods on that node, and the recovery is automatic provided the remaining nodes have capacity. This is the failure class that a cluster tolerates well, and the mitigation is capacity headroom rather than segmentation.

Control plane failure is total for the cluster. If the datastore is unavailable, every write fails, every controller stalls, and every workload continues running but cannot be rescheduled, scaled, or repaired. Running processes keep serving, because the data plane is independent of the control plane, which is why a control plane outage is often survivable for steady-state traffic and fatal for recovery. The mitigation is either a highly available control plane within the cluster or a second cluster whose control plane fails independently.

Tenant misbehaviour is the class that segmentation is most often justified by, and the least often quantified. A tenant can exhaust the control plane’s capacity with a pathological number of objects or a hot loop of writes, exhaust a shared resource that quotas do not cover, or trigger a cluster-wide eviction cascade by filling a node. Quotas and admission bound some of this; they do not bound the aggregate object count, the write rate against the API server, or the number of watches a tenant’s controllers establish.

The three classes have different propagation paths, so a single segmentation decision cannot address all of them. Separating production from non-production addresses the tenant-misbehaviour class and the control-plane-failure class. Adding nodes to a pool addresses the node-failure class. Neither addresses the other.

What the control plane’s capacity actually is

The API server’s throughput is bounded by the datastore’s write latency and by the cost of the operations the API server performs per request, including admission, authorization, and serialization. The scheduler’s throughput is bounded by the number of unschedulable pods and the frequency of rescheduling attempts. The controller manager’s is bounded by the number of objects it must reconcile.

None of these scales linearly with node count, and all of them degrade as the object count grows. The practical implication is that the number of objects per cluster is a real capacity dimension, and it is one that quotas in the usual sense do not limit. A tenant creating a large number of short-lived objects, such as one per pipeline run, can consume control plane capacity without ever exceeding a CPU or memory quota.

The mitigation that works is a bound on object churn per tenant: a rate limit on creates and deletes, enforced by the API server’s priority and fairness mechanism rather than by a quota object. That mechanism assigns requests to flows identified by the requesting identity, gives each flow a share of the server’s concurrency, and queues or rejects requests beyond it. It converts a tenant’s write storm into that tenant’s own latency rather than into everyone’s.

The boundary that must be hard: failure independence

The only separation that reliably buys failure independence is one where no shared component exists. Two clusters in the same region sharing a datastore, a load balancer, a DNS zone, or an identity provider are not independent for failures in those components.

The dependency that is most often overlooked is DNS. Every service lookup and every certificate issuance depends on it. A cluster whose workloads resolve names through a shared zone inherits that zone’s failure modes, and a segmentation plan that separates clusters but not zones has separated the compute and left the resolution path shared. The same is true of the container registry: a cluster that cannot pull images cannot start new pods, and a registry outage makes every cluster that depends on it unable to recover from an unrelated node failure.

The honest formulation of a segmentation plan is a list of components with, for each, whether it is shared across the boundary and what the consequence of its failure is. Compute is usually separable. Storage, identity, DNS, registry, and the deployment pipeline usually are not, and each of those is a path by which a failure crosses a boundary that looks clean on a diagram.

The cost side: operational surface

Every additional cluster multiplies the operational surface. Each has its own control plane version, its own certificate rotation, its own admission webhook configuration, its own policy bundle, and its own set of credentials. The multiplication is not additive in effort: a fleet of twenty clusters at version $v$ and one at $v-1$ is a compatibility matrix, and the platform team must test the path against both.

Three mechanisms keep the multiplication manageable, and all three are about making clusters interchangeable rather than individually administered.

Configuration as a single source of truth, applied to every cluster by the same pipeline, so a policy change is one change rather than twenty. Any per-cluster variation is a declared parameter in that source, not an edit made on the cluster.

Fleet-wide version skew bounded to a small window, with the platform’s own tooling able to report the version of every cluster and refuse to add a workload to one that is out of the window. A cluster that drifts is a cluster whose behaviour is untested.

A single identity plane, so a workload’s identity is valid across clusters and the platform’s authorization policy is defined once. This is also what makes cross-cluster service access tractable: a service in one cluster calling a service in another needs an identity the receiving cluster’s policy accepts, and per-cluster identity systems make that a manual mapping.

The cost that remains even with all three is the loss of statistical sharing. Twenty small clusters each need headroom for a node failure, and the headroom does not pool. A single large cluster absorbs a node failure using capacity that other workloads are not using; twenty clusters each need their own slack, so the aggregate reserved capacity is larger. This is the direct trade: segmentation buys failure independence and pays for it in idle capacity.

The intermediate option: logical separation within a cluster

Between one cluster and many there is a middle design that addresses tenant misbehaviour without multiplying control planes. It uses namespaces with strong policy, dedicated node pools selected by taint, and separate API server priority classes for different tenant tiers.

This works for the misbehaviour class, because quotas, network policy, and priority bound what a tenant can do to its neighbours. It does not work for the control-plane-failure class, because the control plane is still shared. It is therefore the right design when the dominant risk is a noisy or runaway tenant, and the wrong design when the dominant risk is a control plane that fails as a unit.

The distinction is worth stating explicitly in a platform’s own documentation, because “the platform is isolated” is true under one of these designs and false under the other, and the difference only becomes visible during the incident that the design was meant to prevent.

What to measure before deciding

The decision is a bet on which failure class dominates, and the bet can be informed rather than guessed. Three measurements bear on it.

The distribution of API server request latency and error rate by requesting identity shows whether any tenant’s traffic is affecting others. If the top one percent of requesters account for a large share of the server’s concurrency, the misbehaviour class is live.

The duration and frequency of control plane unavailability, however caused, shows the cost of the shared component. A control plane that has never been unavailable for more than seconds does not justify segmentation on its own.

The reserved capacity per cluster shows what segmentation costs. If pools are already at high allocation, splitting them makes the headroom problem worse rather than better, and the segmentation plan needs to account for that before it is adopted.

The relationship between the platform’s promises and its topology is the point where this meets service level objectives as a platform contract: a promise of availability that a single shared control plane cannot keep is a promise the topology has already broken.