Hey everyone, welcome to the twenty-ninth issue of The Main Thread. In this article, we will deal with the unpleasant scenario of downstream services failing in a distributed system.
The subtitle says that it’s a matter of time that downstream services will fail. This is not pessimism but an operational reality. Networks partition, DBs hit connection limits. Third-party APIs go down for maintenance. Memory leaks slowly strangle a service until it stops working.
In a distributed system with dependencies, even if each has 99.9% uptime, the compound availability drops to ~99% (0.999 ^ 10 = 0.9900448802). That’s more than 8 hours of potential failure per year from dependencies alone.
The question that comes is whether the system is able to gradually isolate the failures or lets them cascade into a total outage.
We covered how retries and exponential backoff save us from transient blips in a previous article of this newsletter: Timeouts, Retries, and Deadlines. But when a downstream service is genuinely broken for minutes or hours, retries become part of the problem. Every thread in a retry loop is a thread not serving users. Every retried request to an already struggling downstream service makes it harder for that service to recover.
Luckily, we have circuit breakers to deal with this problem. As you may have guessed it, they originate from electrical engineering where a physical circuit breaker in the fuse box cuts power when it detects a fault. Thus, preventing the entire house to burn down. The software analog does the same. It detects that a downstream dependency is broken, stop sending traffic to it, and fails fast until it recovers.
TL;DR
Circuit breakers have three states: Closed (normal), Open (failing fast), Half-Open (probing recovery)
Failure detection uses count-based or time-window-based thresholds
Recovery happens through half-open probing, not blind re-enable
Circuit breakers complement retries: retry transient errors, trip the breaker on sustained failures
The real value is cascading failure prevention, i.e., one degraded service should not take down the whole system
Three States: A State Machine to Understand
A circuit breaker wraps a call to downstream dependency. It maintains internal state that determines whether to actually make the call or fail immediately.
1. Closed State (Normal)
In this state, the requests flow through to the downstream service normally. The circuit breaker monitors outcomes such as successes, failures, and timeouts. When count of failures cross a configured threshold, the breaker “trips” and becomes “open”.
Closed → default healthy state. The breaker is invisible; it just passes through calls and counts outcomes.
2. Open State (Failing Fast)
In this state, the circuit breaker stops sending requests to downstream services entirely and every call fails with a well-defined error (like CircuitBreakerOpenException). Since no network call is made, no threads get blocked waiting for a timeout.
This is an important point: failing fast is better than failing slowly. A call that fails in 1ms frees the thread to serve another request. On the other hand, a call that hangs for 30s holds that thread hostage, starving thread pool. This eventually causes service to fail too.
The open state has a reset_timeout which is a configured duration after which the breaker transitions to Half-Open to probe whether the downstream service has recovered.
3. Half-Open State (Controlled Recovery)
This state is the most nuanced. Here, a circuit breaker allows a limited number of probe requests pass through. If those requests succeed, the breaker transitions back to the Closed state. If they fail, it remains Open.
This approach is necessary because without it, either we stay open forever (service never recovers even after the downstream is fixed), or transitions directly to closed after the timeout (flood a recovering service with traffic and potentially knock it down again).
Half-open lets us verify recovery under real load before resuming full traffic. It is cautious by design.
Failure Detection: Threshold Strategies
How does a breaker know when to trip? There are two main strategies, each with different tradeoffs.
1. Count Based Thresholds
This is the simplest approach where we maintain a sliding window of the last N calls. If the failure rate in that window exceeds a threshold percentage - the circuit trips.
class CountBasedCircuitBreaker:
def __init__(self, window_size=10, failure_threshold=0.5):
self.window_size = window_size
self.failure_threshold = failure_threshold
self.results = deque(maxlen=window_size)
self.state = "CLOSED"
def record_outcome(self, success: bool):
self.results.append(success)
if len(self.results) == self.window_size:
failure_rate = self.results.count(False) / self.window_size
if failure_rate >= self.failure_threshold:
self.trip()The problem with this approach is that count-based windows are sensitive to traffic volume. During low-traffic period, a small number of failures can trip the breaker prematurely. Two failures in a window of 10 during a quiet night might not indicate a real problem - it might just be noise.
2. Time-Window Based Thresholds
This is more robust because it measures failure rates within a fixed time window (example: last 60 seconds), and only consider the threshold meaningful if a minimum number of calls were made.
from collections import deque
import time
class TimeWindowCircuitBreaker:
def __init__(
self,
window_seconds=60,
failure_threshold=0.5,
minimum_calls=5,
reset_timeout=30
):
self.window_seconds = window_seconds
self.failure_threshold = failure_threshold
self.minimum_calls = minimum_calls
self.reset_timeout = reset_timeout
self.calls = deque() # (timestamp, success)
self.state = "CLOSED"
self.opened_at = None
self.half_open_probes = 0
def _prune_window(self):
cutoff = time.time() - self.window_seconds
while self.calls and self.calls[0][0] < cutoff:
self.calls.popleft()
def should_allow_request(self) -> bool:
if self.state == "CLOSED":
return True
if self.state == "OPEN":
if time.time() - self.opened_at >= self.reset_timeout:
self.state = "HALF_OPEN"
self.half_open_probes = 0
return True # Allow probe
return False # Fail fast
if self.state == "HALF_OPEN":
return self.half_open_probes == 0 # Allow one probe at a time
def record_outcome(self, success: bool):
now = time.time()
self.calls.append((now, success))
self._prune_window()
if self.state == "HALF_OPEN":
if success:
self.state = "CLOSED"
self.calls.clear()
else:
self.state = "OPEN"
self.opened_at = time.time()
return
if self.state == "CLOSED":
total = len(self.calls)
if total < self.minimum_calls:
return
failures = sum(1 for _, ok in self.calls if not ok)
if failures / total >= self.failure_threshold:
self.state = "OPEN"
self.opened_at = time.time()The minimum_calls guard is essential because a 100% failure rate with two calls is not the same as 100% failure rate with 200 calls. Without this floor, the breaker will trip on statistical noise during low-traffic period.
What counts as a failure?
This is a design decision. Common candidates could be:
HTTP 5xx responses
Connection timeouts
Read timeouts beyond a threshold
Exceptions from the downstream client
HTTP 4xx responses (bad request, not found) generally should not count because these are caller errors, not downstream failures.
Integration With Retries and Timeouts
Circuit breakers and retries are complementary. They operate at different timescales and address different failure modes.
Retries handle transient failures: a single failed request due to a momentary network glitch, a brief GC pause on the server, or a resource contention spike. Exponential backoff with jitter spreads retry load to prevent thundering herds.
Circuit breakers handle sustained failures: a service that is down for 10 minutes, a DB under severe load, a dependency that won’t recover until an on-call engineer intervenes.
This combination looks like this:
The key ordering is: don’t retry when the circuit is open. If we retry a request against an open circuit, we are retrying into an immediate failure, wasting time and giving the user a slower error than necessary. Check the circuit first.
Also, be thoughtful about what retry attempts contribute to the circuit breaker’s failure count. Retrying the same logical operation multiple times before the circuit trips can be appropriate - but retrying a hundred times across many callers before the circuit has had a chance to open is exactly the cascading failure we are trying to prevent.
Practical Rule: Let the circuit breaker observe the retry outcomes. If retries exhaust without success, that counts as a failure for circuit breaker tracking purposes.
That way, sustained retry failures will eventually trip the breaker and protect the system.
Production Implementation: Resilience4j
For JVM based systems, Resilience4j is the de-facto standard. It provides circuit breakers, rate limiters, retries, bulkheads, and time limiters in a composable, functional style.
CircuitBreakerConfig config = CircuitBreakerConfig.custom()
.slidingWindowType(SlidingWindowType.TIME_BASED)
.slidingWindowSize(60)
.minimumNumberOfCalls(10)
.failureRateThreshold(50)
.waitDurationInOpenState(Duration.ofSeconds(30))
.permittedNumberOfCallsInHalfOpenState(3)
.recordExceptions(IOException.class,TimeoutException.class)
.ignoreExceptions(BusinessException.class)
.build();
CircuitBreaker breaker = CircuitBreaker.of("payment-service", config);
// Wrap your call
CheckedSupplier<PaymentResponse> supplier = CircuitBreaker
.decorateCheckedSupplier(breaker, () -> paymentClient.charge(request));
Try<PaymentResponse> result = Try.of(supplier)
.recover(CallNotPermittedException.class, ex -> fallbackResponse());Key Resilience4j decisions:
slidingWindowType: ChooseTIME_BASEDfor production systems with variable traffic.COUNT_BASEDis simpler but breaks down under traffic spikes or lulls.permittedNumberOfCallsInHalfOpenState: More than one probe gives you statistical confidence in the recovery signal. Three probes is typical, if two of three succeed, you are probably healthy.recordExceptionsvsignoreExceptions: Be precise here. Resilience4j allows you to whitelist exceptions that count as failures and blacklist those that don't. AValidationExceptionfrom your own code is not a signal that the downstream is down.Fallback handling: The
CallNotPermittedExceptionis what Resilience4j throws when the circuit is open. Always handle it explicitly and return a sensible fallback (a cached response, a default value, or a user-friendly error), not an unhandled 500.
Cascading Failure Prevention: Full Picture
Cascading failures are the real danger in distributed systems. It is one failure cascading into a system-wide outage. Circuit breakers are the primary defence but they are one layer of a larger pattern.
Scenario without circuit breaker
Every caller that hits Service B while it's struggling holds a thread, waiting for the timeout. Thread pools are finite. Once exhausted, the calling service itself stops responding because all its capacity is consumed waiting on a broken dependency.
Scenario with circuit breaker
The failure is contained. Service B being down degrades the feature but doesn't take down Service A or the API Gateway.
Bulkheads: defence in depth
Circuit breakers prevent us from talking to a broken service. Bulkheads prevent a broken service from consuming all our capacity. This pattern is borrowed from ship design and it isolates thread pool per dependency.
ThreadPoolBulkheadConfig bulkheadConfig = ThreadPoolBulkheadConfig.custom()
.maxThreadPoolSize(10)
.coreThreadPoolSize(5)
.queueCapacity(20)
.build();
ThreadPoolBulkhead bulkhead = ThreadPoolBulkhead.of("payment-service", bulkheadConfig);Without bulkheads, a slow downstream can consume our entire thread pool. With bulkheads, it can only consume the 10 threads allocated to it, leaving the other 90 to serve other requests.
Combine circuit breakers with bulkheads for proper isolation: the circuit breaker cuts off traffic when the dependency is broken; the bulkhead limits how much damage a slow-but-not-broken dependency can do.
Timeouts: non-negotiable baseline
Circuit breakers require timeouts to function correctly. If downstream calls never time out, a hanging service will hang your threads forever, thus, the circuit breaker can't record the outcome and can't trip.
Set aggressive timeouts for downstream calls. A service that usually responds in 50ms probably shouldn't have a 30-second timeout. A 500ms or 1-second timeout forces failures to surface quickly, makes them visible to the circuit breaker, and keeps your thread pool healthy.
The practical rule: timeout < retry budget < circuit breaker window. The timeout should be tight. The retry budget (max time to spend retrying) should be within acceptable user-facing latency. The circuit breaker window should be long enough to distinguish sustained failures from transient ones.
Observability
If we can’t observe a circuit breaker, it becomes a liability. We need to know:
Current state per circuit: Is payment-service’s circuit breaker open right now?
Trip frequency: How often does each circuit trip per day?
Failure rate trends: Is the failure rate trending up towards threshold?
Half-open probe outcomes: Are recoveries succeeding or bouncing back to open?
Resilience4j exposes these metrics via Micrometer, which feeds into Prometheus/Grafana. For custom implementations, we should emit these as counters and gauges to whatever observability stack we use.
Alert on circuit breakers transitioning to Open. This is an operational signal: a dependency is down and the system is degrading. It might not be a P1 (if the fallbacks are good), but it always warrants investigation.
When Not to Use Circuit Breakers
Circuit breakers add complexity. Not every call needs one.
Skip circuit breakers for:
Calls to our own in-process code
Calls to local cache (Redis on the same host, in-memory cache)
Non-critical paths where failure is already the expected result
Calls so infrequent that the circuit breaker's statistical window never fills meaningfully
Always use circuit breakers for:
Synchronous calls to external services (payment providers, notification services)
Database calls in services where DB availability is not assumed
Any dependency where a slowdown would cause thread pool exhaustion in your service
Any dependency that has historically had availability issues
The pattern pays for itself in proportion to our dependency count and traffic volume.
Mental Model to Carry Forward
Distributed systems fail partially, and partial failures are the hardest to handle. A service that's completely down is easy: everything fails fast and we know what's wrong. A service that's slow, intermittently failing, or failing for 30% of requests is much harder.
Circuit breakers give us a structured, stateful mechanism for handling that middle ground. They encode operational knowledge (this service is failing) into the execution path (therefore stop calling it), with a principled recovery path (probe it carefully, restore traffic when it's healthy).
Retries handle the blips. Circuit breakers handle the outages. Timeouts make failures explicit. Bulkheads contain the blast radius. Together, these patterns form the fault-tolerance layer that separates systems that stay up from systems that cascade into full outages at the worst possible moment.
We must build these in early. They are much harder to retrofit into a system that's already under pressure.
I hope you liked this issue and I would love to know your thoughts on this.
Namaste!
— Anirudh
