Blast Radius and Cell-Based Isolation

Availability arithmetic describes how often a system fails. Blast radius describes how much fails at once, and the two are independent: a system can be unavailable 0.01% of the year in one event that takes every customer offline, or the same 0.01% spread across single-tenant glitches that nobody outside the affected account notices. The first is a business catastrophe reported as an excellent availability number.

Blast radius is measured as the share of traffic or the set of tenants affected by one failure. It is designed, not inherited, and the design work consists almost entirely of finding the places where otherwise independent things share a resource.

Sharing is the propagation path

Failures do not spread through service boundaries. They spread through shared resources, and the boundaries only make the sharing harder to see.

A connection pool sized for steady-state traffic is a shared resource that a single slow dependency can exhaust. Once the pool is empty, every request that needs it queues, including requests for unrelated functionality that happens to borrow the same pool. The same shape appears with thread pools, file descriptors, memory, and any semaphore that is acquired per request.

An in-process cache with one hot key is a shared resource. Every request for that key hits the same upstream at the same moment, so the cache turns uniform load into a spike and provides no smoothing at all. A shared DNS resolver, a shared configuration service, a shared certificate, and a shared secrets store are all single points through which every service’s requests pass at startup or on refresh.

Deployment is the most overlooked shared resource. If a fleet is updated by one rollout that replaces all instances in one step, a bad build is a total outage regardless of how many availability zones the instances span. The release discipline that lets a bad build be caught before it reaches the whole fleet is a compatibility question as much as a deployment one, taken up in Independent Deployability and Cross-Service Contracts.

The bulkhead pattern addresses the first group. Resources are partitioned by dependency or by tenant so that exhaustion in one partition cannot consume another’s. Two pools of ten connections behave very differently from one pool of twenty when one dependency hangs: the first keeps the healthy dependency serving while the second starves it. The cost is idle capacity, since each partition is sized for its own peak, and queues form earlier because each pool is smaller. Bulkheads are cheap to add inside a process and are the first isolation tool to reach for.

They are not sufficient. A bulkhead cannot protect against a bad deploy, a corrupt configuration push, or a poison message that crashes the process, because those take out every instance at once.

Cells as the unit of containment

A cell is a complete, independent instance of the serving stack, including its own data store, deployed and operated separately from its siblings. A routing layer in front maps each request to exactly one cell, so a failure inside a cell is confined to the traffic assigned to it. With ten cells, a total cell failure costs a tenth of the traffic, and the remaining nine continue serving their own tenants with no shared state to coordinate.

The design rule that makes cells work is that nothing in the request path may be shared across them. Shared databases, shared caches, shared queues, and shared authentication services each reintroduce a global failure domain, and a cell architecture that keeps any of them gains nothing but operational overhead. A shared database is the most complete defeat of isolation, because it is also the reason the two halves cannot be separated in the first place, as discussed in Drawing Service Boundaries Around Consistency Requirements.

Cell size is a direct trade between blast radius and fixed cost. Smaller cells reduce the fraction lost per failure but multiply the number of independent stacks to patch, monitor, and capacity-plan, and they make cross-cell operations harder because every global query becomes a fan-out. A common arrangement is a cell count in the low tens: enough that one loss is a rounding error in aggregate availability, few enough that the fleet is still operable as a small number of units.

Capacity is where cell design becomes arithmetic rather than taste. If each cell runs at utilisation u and one cell is lost, the survivors must absorb the load at u·N/(N−1). Ten cells at 60% become 66.7% after a loss, which is comfortable; four cells at 80% become 106.7%, which is not possible, so the fourth cell’s traffic has nowhere to go and the failure cascades. The formula sets a hard ceiling on utilisation for a given cell count, and it is the reason cell counts are chosen together with a target post-failure utilisation rather than in isolation.

The alternative to absorbing a lost cell’s traffic is to fail it, leaving those tenants down until the cell returns. That is a legitimate choice for a multi-tenant system where tenants are independent, and it makes capacity planning trivial. It is not acceptable when the system has a single dominant tenant, because one cell then holds most of the traffic.

Shuffle sharding and tenant placement

Assigning tenants to cells one-to-one gives each tenant a single cell, so a cell failure takes out every tenant in it. Assigning tenants to combinations of cells with overlap gives each tenant a set of cells and spreads the risk. Shuffle sharding formalises this: each tenant is deterministically assigned k of N cells, and requests may be served by any cell in the tenant’s set.

The isolation comes from the number of distinct assignments. With N = 8 and k = 2 there are C(8,2) = 28 possible sets, so a failure affecting two specific cells damages the tenants assigned exactly that pair, roughly 1/28 of the assignment space. Single-cell assignment with N = 8 has only eight possible sets, and a failing cell damages 1/8 of tenants. The improvement compounds as N grows: C(16,2) = 120, C(32,2) = 496.

The cost is that a tenant’s requests may land on any of k cells, so either the tenant’s data is replicated to all of them or the router sends each request to the one cell holding that tenant’s data, which reduces the scheme to placement with a fallback set.

import hashlib

def cells_for(tenant: str, cells: list[str], k: int = 2) -> list[str]:
    """Deterministic k-of-N assignment; stable as long as the cell list is stable."""
    ranked = sorted(
        cells,
        key=lambda c: hashlib.sha256(f"{tenant}/{c}".encode()).digest(),
    )
    return ranked[:k]

Determinism matters more than balance. A router that recomputes assignments from a live membership list will move tenants whenever a cell is added or removed, and moving a tenant means migrating its data while it is serving traffic. Assignment changes are therefore deliberate operations, not consequences of autoscaling.

The routing layer is a shared dependency

Every cell architecture has one component that is not a cell, and it is usually the weakest part of the design. The router holds the tenant-to-cell map, terminates TLS, and applies policy. It is shared by definition, so its failure is global.

The usual mitigations are to keep the map small and cache it at the edge, to make the router stateless with the map distributed as a versioned artefact, and to accept that a stale map is better than no map: routing a moved tenant to its old cell produces errors for that tenant, while failing to route produces errors for everyone.

Control planes deserve the same scrutiny. Service discovery, configuration distribution, secret rotation, and certificate issuance are typically global even when the data plane is cellular. A control-plane outage that prevents new instances from starting is survivable; one that prevents existing instances from refreshing credentials on a timer is not, because it converts a control-plane problem into a simultaneous data-plane failure across every cell at once. Long cache lifetimes and stored credentials with overlapping validity turn that cliff into a slope.

Measuring whether the isolation holds

Isolation claims are testable, and the test is not a diagram review. The direct check is to remove one cell from service and observe what happens to the traffic assigned to it and to the rest of the fleet. If the remaining cells absorb the load without exceeding their post-failure utilisation, and the affected tenants fail cleanly rather than hanging, the boundary works.

Three observations falsify most designs. Latency in unaffected cells rises when one cell is lost, which means a shared resource exists somewhere in the path. A cell failure produces errors in tenants that are not assigned to it, which means the routing map leaks. Recovery takes longer than the failure itself, which usually means the router is retrying into a cold cell and amplifying the load, or that cell data has to be restored before the cell can serve.

None of this replaces availability arithmetic. A cell failure is still a failure, and if every tenant is in exactly one cell then a cell outage is an outage for those tenants regardless of how well the boundary held. Cells bound the damage; they do not remove it, and the remaining exposure is managed by how cells are assigned, how much traffic each holds, and whether the system can move tenants out of a degraded cell before it fails completely.