Cell-Based Isolation and the Blast Radius of a Bad Deployment

A regional deployment with one of everything is a single unit of failure. A bad configuration change, a memory leak that appears only under a particular query pattern, or a data corruption bug reaches every tenant at once, because there is no boundary between them. Multi-region topology does not fix this: three regions running the same broken build fail in three places. The boundary that limits blast radius is horizontal, not geographic, and it is created by splitting the regional deployment into independent cells.

Blast radius is a fan-out property

The size of an incident is determined by how many distinct things share a failure domain. In a monolithic regional stack, the failure domain is the stack, and its size is the whole tenant population. A cell architecture divides that population into groups, each served by its own instance of the stack, so a fault in one cell reaches only the tenants assigned to it.

The isolation is only real if the cells share nothing that can propagate a fault. Shared dependencies that routinely break this:

• A single database instance for all cells, with cells as separate application pools. The application fault is contained; the database fault is not.
• A shared cache cluster. A cache is usually the first thing to become a bottleneck, and a shared one turns a per-cell load spike into a global one.
• A shared queue or topic with a single consumer group, where a poison message blocks every cell’s consumer.
• A shared control plane that is in the request path — a feature-flag service called per request, a shared authorization service, a global rate limiter.
• A shared deployment pipeline that pushes to all cells simultaneously, which converts a bad build into a full-fleet outage regardless of how well the runtime is partitioned.

The last one is the most commonly overlooked and the easiest to fix: deploy to one cell, observe, then proceed. The observation window is a policy parameter, and its length trades deployment speed against the size of the blast radius when the build is bad.

Cells are not availability zones

An availability zone is a failure domain defined by the infrastructure provider: separate power, cooling, and network, with low latency between zones in the same region. A cell is a failure domain defined by the application: a subset of tenants served by a complete, independent copy of the application stack. The two are orthogonal.

A common arrangement is one cell per availability zone, which gives both infrastructure isolation and application isolation, but it forces the cell count to match the zone count — usually three — and three cells is too few to contain a tenant population of any size. A cell can also span zones: a cell with its own database replicated across three zones is resilient to a zone failure while still isolating its tenants from other cells’ faults. That is usually the better trade, because zone failures are rare and correlated-fault events like a bad query plan are common.

The number of cells follows from the blast-radius target. If the acceptable fraction of tenants affected by a single-cell fault is 5 percent, at least 20 cells are needed, and more if the tenant distribution is uneven and the largest cell is allowed to exceed the average.

Routing tenants to cells

Every request must be routed to the cell that serves its tenant, which means the routing layer needs a tenant-to-cell mapping. The mapping is small — one entry per tenant — and it is read on the request path, so its availability matters as much as the cells’.

Three properties are required of the map. It must be stable, so a tenant does not move between cells on every deploy. It must be recoverable, so a routing layer restart does not lose the assignment and reshuffle tenants. And it must be spread, so that a single cell’s failure does not take out a disproportionate share of traffic.

A hash of the tenant identifier satisfies stability and spread but not balance: with 20 cells and a few very large tenants, the hash can place several large tenants in the same cell. Explicit assignment satisfies balance and costs a lookup. The usual compromise is a hash into a large number of buckets, with buckets assigned to cells and reassigned by hand when a cell grows too large.

# bucketed assignment: stable for a tenant, adjustable per bucket
bucket = fnv1a64(tenant_id) % 1024
cell   = bucket_map[bucket]      # 1024 entries, reassignable without moving tenants

# capacity check before reassignment
if cell_load[target] + bucket_weight[bucket] > cell_capacity:
    refuse the move

The bucket count is the granularity of rebalancing. With 1,024 buckets and 20 cells, each bucket is roughly 0.1 percent of the tenant population, so a cell can be grown or shrunk in increments of that size. Too few buckets makes rebalancing coarse; too many makes the map large and the reassignment frequent.

The cell map as a dependency

The map is a dependency on the request path, and a dependency on the request path can fail. If the map is unavailable, every request fails, which defeats the purpose of isolation. Two mitigations are standard: cache the map in the routing layer with a long TTL and serve stale on failure, and make the map’s read path independent of the cells it describes. A map service hosted inside cell 7 cannot be used to route requests to cell 7.

A third mitigation is to make the map derivable. If the assignment is a pure function of tenant identifier and a versioned assignment table, the routing layer can compute it locally from a table it already has, and the table’s update is the only coordination needed. That is the bucketed scheme above, and it is why the scheme is worth the extra indirection.

Shuffle sharding and the isolation of noisy tenants

Even with cells, a cell full of tenants is a shared fate for those tenants. Shuffle sharding reduces the effective overlap between any two tenants by giving each tenant a subset of the available resources rather than one dedicated resource.

The construction: for each tenant, choose k resources at random from a pool of n, and route all of that tenant’s requests only to those k. A tenant’s traffic is spread across its k resources, and a resource failure affects only the tenants whose subset includes it. With n = 8 and k = 2, each resource appears in a tenant’s subset with probability k/n = 0.25, so a single resource failure affects about a quarter of tenants rather than all of them, and two simultaneous failures affect only tenants whose subsets include both — a much smaller fraction, since the subsets are chosen independently.

The numbers improve quickly with pool size. With n = 20, k = 2, one failed resource affects 10 percent of tenants; two failed resources affect roughly 0.5 percent, because the chance that a tenant’s two-resource subset contains both failed resources is about (k/n)². That quadratic falloff for correlated failures is the property that makes shuffle sharding worth the routing complexity: the common case of a small number of overloaded or broken resources degrades a small, shrinking fraction of tenants rather than a fixed fraction.

The cost is that each tenant uses only k of n resources, so the pool must be sized for the peak load of the largest subset, not the average. Effective utilization is lower than a fully shared pool, and the gap is the price of isolation.

Cells and shuffle sharding together

The two techniques compose. Cells provide coarse isolation at the deployment level — a bad build, a schema migration, a corrupted index. Shuffle sharding provides fine isolation inside a cell — a hot shard, a slow disk, a noisy neighbour. A typical arrangement is a small number of cells, each containing a pool of shards that tenants are assigned to by shuffle sharding.

The failure that this composition handles well is the slow resource. A single shard whose disk is degraded will serve its assigned tenants slowly, and those tenants’ requests will time out and retry. Retries amplify the load on the degraded shard, which is the cascading failure pattern described in the SRE book: a component near its limit receives more work precisely because it is slow. Shuffle sharding limits the amplification to the tenants on that shard, and the retry budget must be limited too, or the amplification spreads to healthy shards through shared queues.

What a cell architecture does not fix

Three classes of failure cross cell boundaries regardless of how the cells are built.

Data corruption in a shared upstream. If all cells read from a common source of truth that is corrupted, every cell serves the corrupted data. Cells isolate compute and availability, not correctness of shared data.

Client-side failures. A client library with a retry storm, or a single client sending pathological requests, can consume shared front-end capacity before any cell sees the request. The front door — the load balancer and the TLS termination — is often the least partitioned part of the system, and it is where the blast radius is largest.

Identity and authorization. These are usually global because a user must be recognized everywhere. A failure in the identity service affects all cells, and a change to it is a full-fleet change. The mitigation is to keep the per-request path dependent on a locally verifiable token rather than a live lookup, which is the arrangement described in pinning data to a region; it converts a shared-dependency failure into a staleness window.

The honest summary is that cells reduce the blast radius of the faults that occur inside the cell — code defects, resource exhaustion, bad configuration — and they do nothing for the faults that occur in the parts deliberately left shared. The design work is deciding which parts those are, and making the list short and deliberate rather than emergent.

The operational cost

Cells multiply everything that is per-cell: deployments, dashboards, alerts, capacity reviews, and schema migrations. A migration that must run in 20 cells is 20 times the operational surface, and running it in parallel recreates the shared-failure problem the cells were meant to solve.

The usual resolution is a migration runner that advances cell by cell with a gate between them, and a schema that tolerates both old and new versions during the rollout. That is the same expand-and-contract pattern used for zero-downtime migrations in a single database, applied at the cell level. Without it, the cells become a fleet of snowflakes whose schemas drift, and the isolation becomes a reason not to upgrade rather than a reason to be safe.

Cell count is therefore a real trade against operational capacity, and it is one of the few architecture decisions where a smaller number is often correct. The relevant question is not “how many cells can be built” but “how many cells can be deployed, migrated, and observed by the team that owns them”, because a cell architecture that cannot be operated safely under change has moved the risk rather than reduced it.