Bulkhead Isolation and the Design of Partial Degradation

A ship’s hull is divided into compartments so that a breach floods one of them rather than the whole vessel. The software analogue is capacity partitioned by dependency, so that a single slow or failing dependency exhausts its own allocation and leaves the rest of the service able to work.

The mechanism is trivial: separate pools, separate limits, separate queues. The design problem is the allocation. Every partition reserves capacity that the others cannot use, and a partition sized for a dependency’s worst case is idle in its normal case. The whole subject is the management of that reserve.

The failure that bulkheads prevent

Without isolation, one dependency’s latency determines the service’s concurrency. A service with a shared pool of two hundred workers, calling four dependencies, sends every request through the same pool. If one dependency’s latency rises from 20 milliseconds to two seconds, every request that touches it holds a worker for two seconds, and after a few seconds all two hundred workers are held by requests waiting on that one dependency. Requests that do not touch it at all cannot get a worker, because there are none.

The failure is not that the slow dependency is slow. It is that the slow dependency’s cost is denominated in a resource shared with everything else. Isolation changes the denomination: the slow dependency’s cost is denominated in its own partition, and when that partition is full, requests that need it fail while requests that do not need it proceed.

The same argument applies at every level of the hierarchy: between dependencies within a service, between tenants within a multi-tenant service, between request classes with different cost profiles, and between a service’s own foreground work and its background work. The partition boundary is chosen where the failure modes are least correlated.

Sizing a partition

A partition’s size follows from the concurrency that the dependency needs at peak, which is the Little’s Law product for that dependency alone: arrival rate to the dependency multiplied by its latency at the target percentile. A dependency receiving 300 calls per second with a 30-millisecond latency needs 9 concurrent calls. Sized for the p99 latency instead of the mean, it needs whatever the p99 implies, which is typically two to three times larger.

The partition is not the same as the queue. A partition of 9 with no queue sheds immediately when the tenth concurrent call arrives, which is a tight bound and produces rejections on any variance. A partition of 9 with a small queue absorbs variance at the cost of latency for the queued calls. The combination of the two is the admission decision described in Bounded Queues and Load Shedding at the Admission Boundary, applied per partition.

The sum of the partitions is the constraint. If four dependencies each need 50 concurrent calls at peak and the service can support 120, the partitions cannot all be sized for simultaneous peak. Two resolutions: size for the peak of the sum rather than the sum of the peaks, which is smaller because peaks rarely coincide, or accept that the service sheds when several dependencies peak together. The first is a statistical bet that needs a measured correlation, and the second is an explicit statement that the service is provisioned below its worst case.

What happens when a partition is full

A full partition has to produce an answer, and the answer determines whether the isolation helped. Four options, in order of preference for a read path.

Serve a fallback. The feature returns a default, a stale value, or an empty result. This requires that the caller have a defined degraded behaviour, which is a product decision that has to be made before the incident. A recommendation panel that renders empty is a degraded page; a checkout that renders empty is a broken one.

Serve from a cache with a stated freshness bound. The bound is the contract: a cached price older than the bound is wrong, not degraded. The cache’s own eviction and the partition’s fullness interact, because a cache that is being refreshed through the failing dependency stops refreshing, and the entries age past their bound while still being served.

Fail the request with a distinguishable error. The distinction matters because a generic error is retried by every layer above, which reinstates the load. A specific “dependency unavailable” error that the retry policy treats as terminal stops the amplification.

Block until the partition frees. This is the option that defeats the purpose. Blocking converts the partition back into a shared resource, because the blocked requests hold whatever else they hold while they wait.

The choice is per call site, not per dependency. The same dependency may be essential for one endpoint and optional for another, and the partition’s fullness has different consequences in each.

Degradation as a designed state

Partial degradation only works if the degraded path exists in the code before it is needed. A fallback written during an incident is untested, unmeasured, and usually slower than the path it replaces.

The design questions are concrete. Which features can be dropped entirely, which can return stale data, which can return a reduced result set, and which must fail? What does the page look like when each of those happens, and is that acceptable to the people using it? What is the latency of the degraded path, and does it actually help, or does the fallback itself call a dependency that is also degraded?

The last question catches a common error: a fallback that queries a secondary store which is itself under the load that the primary just shed. The secondary becomes the new bottleneck, and the fallback’s capacity has to be part of the sizing. A fallback used only during failure is sized for the failure rate of the primary, which is the full traffic rate, so the secondary needs capacity comparable to the primary. That is rarely affordable, and it is why fallbacks are usually cheap defaults rather than alternative data sources.

Isolation is not redundancy

A bulkhead limits the blast radius. It does not restore the lost function. If the dependency is the only source of the data, isolating its failure means the feature is unavailable and the rest of the service works. Making the feature available requires a replica, a cache, or a degraded mode, and those are separate mechanisms.

The confusion is common in incident reviews: a service with well-partitioned pools still fails the requests that needed the broken dependency, and the isolation is credited with a recovery it did not provide. What the isolation provided is that the requests which did not need it continued to succeed, which is a different and often larger benefit.

The same distinction applies to the circuit breaker, which stops calls to a failing dependency without providing an alternative. Circuit Breaker State Machines and Half-Open Probing covers the state machine; the bulkhead and the breaker are complementary in that the breaker reduces the rate of failed calls while the bulkhead bounds the resources those calls consume before the breaker opens.

The interaction with concurrency limits

A per-partition concurrency limit and an adaptive global limit can conflict. The global limit shrinks when latency rises, and it shrinks by reducing admissions across all partitions, including the healthy ones. If the global limit is enforced before the partition is chosen, a slow dependency reduces the concurrency available to fast ones, which is precisely the coupling the bulkhead was meant to break.

The correct ordering is to partition first and limit within the partition. The global limit then applies to the aggregate, as a second line of defence, and it should be sized so that it does not bind before the partitions do. When it does bind, the isolation is nominal: the partitions exist but the shared ceiling is the real constraint. Concurrency Limits Derived from Little’s Law describes how the limit is derived and why a limit that reacts to aggregate latency cannot distinguish a slow partition from a busy service.

Threads, connections, and the cost of partitions

A partition needs its own execution resource, and the resource type determines the cost.

Separate thread pools per dependency are the traditional form. Each pool costs its threads’ stacks, which is on the order of a megabyte of virtual address space each with default settings, and the scheduler’s cost of context switching between them. With twenty dependencies the thread count is in the hundreds, and the memory is real even when the threads are idle.

Separate connection pools are cheaper and bound a different resource. A dependency’s connection pool limits concurrent calls at the transport, which is exactly the resource that a slow dependency consumes. The pool’s size is the partition, and its queue is the buffer. This is the form most HTTP clients and database drivers support natively.

Semaphores are the cheapest form: a counter with no dedicated threads, where the calling thread blocks or fails. Go’s buffered channel used as a semaphore, or a counting semaphore in any language, costs a counter and a wait queue. The advantage is that the partition can be sized independently of any pool, and the disadvantage is that the blocked caller still occupies whatever it occupied before it blocked.

Asynchronous runtimes blur the distinction, because a “worker” is a task rather than a thread and its cost is a small allocation. The partition is still meaningful: it bounds the number of concurrent operations against the dependency, which is the quantity that matters, regardless of what holds them.

type bulkhead struct{ sem chan struct{} }

var errBulkheadFull = errors.New("bulkhead full")

// do admits fn if a slot is free; otherwise it fails without queueing.
func (b *bulkhead) do(ctx context.Context, fn func(context.Context) error) error {
	select {
	case b.sem <- struct{}{}:
		defer func() { <-b.sem }()
		return fn(ctx)
	default:
		return errBulkheadFull
	}
}

The default branch is the policy choice. Waiting on ctx.Done() as an alternative to the semaphore would make the bulkhead a queue bounded by the caller’s deadline, which is often a better trade than immediate shedding, because a request with budget left can afford to wait and one without cannot. Selecting on the deadline rather than on a fixed wait converts the partition’s queue depth into time, which is the unit the caller cares about.

Where partitioning fails

Partitions multiply the state that has to be tuned. Twenty dependencies with twenty limits, twenty queues, and twenty sets of thresholds is twenty places to be wrong, and the wrong ones are found during incidents. Deriving the limits from measured concurrency rather than from constants reduces the number of independent decisions, and grouping dependencies with similar latency and criticality reduces the number of partitions.

Partitions also fragment capacity. A partition that is idle cannot serve a neighbour that is full, so the total capacity in use is lower than the total allocated. The utilization penalty is the price of isolation, and it grows with the number of partitions and the variance of their load. Measuring the peak-to-mean ratio per partition gives the size of the penalty; a partition with a flat load wastes nothing, and one with a bursty load wastes most of its allocation most of the time.

The final limitation is that partitioning assumes the failure is confined to a dependency. A failure in a shared resource that every partition uses, such as the runtime’s scheduler, the host’s network stack, or a global lock, crosses every partition boundary. Isolation at the dependency layer cannot isolate a shared layer, and the boundary has to be placed above whatever is shared. Metastable Failure and the Load That Keeps a System Down is the case where the shared resource is the feedback loop itself, and no partition contains it.