Circuit Breaker State Machines and Half-Open Probing

A caller that keeps sending requests to a dependency which is failing spends its own resources on calls that cannot succeed. The call has to be made, the connection attempted, the timeout waited out, the error parsed and handled. At a one-second timeout and a hundred requests per second, a broken dependency can occupy a hundred concurrent slots indefinitely, and those slots are the caller’s capacity, not the dependency’s.

A circuit breaker replaces that with a local decision: after a threshold of failures, stop calling for a while, and fail immediately instead. The saved resource is the caller’s time, and the secondary benefit is that the dependency receives no traffic while it recovers.

Three states and the transitions between them

The machine has three states and is deliberately small.

Closed is normal operation. Calls pass through, and the breaker counts outcomes in a rolling window. When the failure rate in that window crosses a threshold, the breaker transitions to open.

Open rejects calls without attempting them. The rejection is immediate, which is the point: the caller’s thread or goroutine is freed in microseconds rather than after a timeout. The state has a dwell time, after which the breaker transitions to half-open.

Half-open permits a limited number of trial calls. If they succeed, the breaker closes. If any of them fails, it reopens and the dwell timer restarts.

The transition on failure is the one that carries most of the design. A half-open breaker that returns to open on a single failed probe is conservative: one unlucky request keeps the dependency cut off for another full dwell period. A breaker that requires a probe success ratio is more forgiving but admits more failures to real traffic. Both are defensible; what is not defensible is a half-open state with an unlimited probe count, because that is equivalent to closing the breaker at the first probe with no evidence.

The probe count is a budget, not a rate. If a recovering dependency can serve ten requests per second and the breaker releases a hundred probes at once, the recovery fails under the probe load and the breaker reopens, having caused the failure it was testing for. The probe count should be small in absolute terms, on the order of one to five, and spaced by the same backoff reasoning that governs retries.

Counting failures in a window

A naive counter that increments on failure and resets on success never opens under a mixed workload: an occasional failure followed by an occasional success keeps the counter low. The threshold has to be a ratio over a bounded population.

Two window shapes are in common use. A count-based sliding window keeps the last N outcomes in a ring buffer and computes the failure ratio over them; it adapts to traffic volume, opening quickly under high load and slowly under low load. A time-based window buckets outcomes by second and sums the last N buckets, which adapts to time rather than volume and behaves consistently regardless of request rate.

Both need a minimum population. A breaker that evaluates a ratio over three requests opens on a single failure at thirty-three percent, which is noise. The minimum should be large enough that one failure cannot cross the threshold, so a fifty percent threshold needs at least four samples and a practical floor is well above that.

The failure classification matters more than the threshold. A connection refused, a timeout, and a 500 are dependency failures. A 400 caused by the caller’s own malformed request is not: the dependency answered correctly and quickly, and counting it as a failure opens the breaker against a healthy service. A 404 is likewise a valid answer. The set of counted outcomes should be exactly those that indicate the dependency could not do its job, plus the ambiguous ones like timeouts where the dependency may have processed the request. The ambiguity of timeouts has a second consequence, covered in Cancellation Propagation and the Work That Outlives Its Caller: a timed-out call may still be executing at the dependency, so a breaker that counts timeouts is also protecting the dependency from work it has already accepted.

def allow(self) -> bool:
    now = monotonic()
    if self.state == "open":
        if now - self.opened_at < self.dwell:
            return False
        self.state = "half_open"
        self.probes = 0
    if self.state == "half_open":
        if self.probes >= self.max_probes:
            return False          # trial budget exhausted; caller fails fast
        self.probes += 1
    return True

The check runs before the call, so the state transition happens on the first call after the dwell expires rather than on a timer. That is a deliberate simplification: it avoids a background thread per breaker and makes the behaviour deterministic under test. The cost is that an idle breaker stays open until traffic returns, which is harmless.

What the caller sees when the circuit is open

An open breaker produces an error that the caller did not receive from the dependency, and the difference matters for every layer above. Three treatments are possible, and the choice is a product decision as much as a reliability one.

Failing fast with the same error class as a dependency failure keeps the caller’s error handling uniform. It also means a retry policy upstream will retry, which defeats the breaker entirely: the breaker exists to stop calls, and an upstream retry layer that ignores the local failure reinstates them. The breaker’s rejection needs to be distinguishable, either as a distinct exception type or as a response that the retry policy treats as terminal.

Returning a degraded response requires the caller to have one. A recommendation service that cannot reach its model can return an empty list, and the page still renders. That is only correct when the degraded response is genuinely acceptable, which is a property of the feature rather than of the breaker. Bulkhead Isolation and the Design of Partial Degradation treats the design of those fallbacks.

Returning a cached value is a third option, and it is the most dangerous without a freshness bound. A cached response served for an unbounded period converts a dependency outage into a correctness problem that surfaces much later.

Per-target, not per-service

A breaker keyed on the service name opens for every instance when one instance fails, which removes healthy capacity exactly when the system needs it. A breaker keyed on the individual host or endpoint isolates the failure to the target that produced it, and a client with several targets continues to use the healthy ones.

The trade-off is cardinality. A client with a hundred targets has a hundred breaker states, and the failure signal for any single one arrives at a hundredth of the request rate, so each breaker needs a longer window to accumulate a usable sample. This is why per-target breakers are usually paired with outlier detection at the load balancer, where the ejection decision is made on connection-level signals rather than on request outcomes.

The middle ground is a two-level scheme: per-target breakers with generous thresholds, plus an aggregate breaker that opens when a large fraction of targets are failing. The aggregate catches the case where the failure is not in any single target, which is the case a per-target scheme cannot see.

Recovery is the hard part

Closing a breaker is a decision to send real traffic to a dependency whose health is unknown, and every parameter in the design is a guess about how much traffic that dependency can absorb while it recovers.

The failure mode is a thundering herd at the moment of closing. If a hundred callers each hold an open breaker against the same dependency, and all of them transition to half-open at the same dwell expiry, the dependency receives a hundred probe bursts simultaneously. Each caller’s probes are individually small and collectively a load spike. Randomising the dwell per client, or adding jitter to the transition time, spreads the recovery across a window rather than an instant.

A second failure mode is that the dependency’s own recovery is slower than the dwell. A dependency that needs minutes to rebuild a cache or reconnect a pool will fail every probe for several dwell periods, and each probe burst restarts whatever recovery it had begun. The breaker cannot know this, so the dwell has to be long enough to cover the plausible recovery time of the things being called. That is a per-dependency constant, and setting one global value for all dependencies guarantees it is wrong for most of them.

A third is that the breaker and the dependency’s own admission control disagree. The dependency sheds load with a 503, the breaker counts it as a failure and opens, and the dependency is now receiving no traffic at all rather than the fraction it could have served. A dependency that sheds should signal the difference between “overloaded, back off” and “broken, stop”, and the breaker should treat the first as a reason to slow down rather than to stop. Retry-After and the 429 versus 503 distinction exist for this purpose, and a breaker that ignores them is measuring the wrong thing.

Where breakers do not help

A breaker protects against a dependency that fails fast enough to be detected. A dependency that responds slowly, at just under the timeout, never trips a failure threshold while consuming the caller’s full concurrency budget. The latency is the failure, and no error counter sees it. Detecting that case requires a latency-based threshold, which is a different mechanism with its own tuning problem, and it belongs with the concurrency limits described in Concurrency Limits Derived from Little’s Law.

A breaker also does nothing for a dependency that is a single point of failure with no alternative. Opening the circuit converts a slow failure into a fast one, which is an improvement in the caller’s resource usage and no improvement in the outcome. The outcome requires redundancy, and that is a different layer.

The final limitation is that the breaker’s state is local knowledge. Ten clients each with their own breaker make ten independent decisions from ten partial views, and the aggregate behaviour is whatever those decisions sum to. Sharing the state through a coordination service is possible and introduces the coordination service as a dependency of the mechanism that exists to survive dependency failure.