The Circuit Breaker Pattern and Bulkhead Isolation
Stop cascading failures by tripping circuits and isolating resource pools per dependency.
Why Cascading Failures Happen
In a microservice backend, your Node.js API often depends on downstream services: a payments provider, an inventory service, a database. When one dependency becomes slow (not even down — just slow), every request that touches it piles up.
- Each pending request holds an event-loop slot, a socket, and memory.
- Callers retry, multiplying load on the already-struggling dependency.
- Your healthy endpoints starve because the process is saturated waiting on the sick one.
This is a cascading failure: one slow dependency drags down the entire service. Two patterns fight this — the circuit breaker (stop calling a failing dependency) and the bulkhead (cap how many resources any one dependency can consume).
The Naive Retry Makes It Worse
A common first instinct is to wrap a flaky call in a retry loop. But blind retries amplify a partial outage into a full one — you triple the traffic at the exact moment the dependency can least handle it.
Below, a fragile downstream is hammered. Notice how retries turn 1 logical request into many physical calls.
async function callDependency(attempt) {
// Simulate a dependency that fails 70% of the time
if (Math.random() < 0.7) {
throw new Error('dependency timeout');
}
return 'ok';
}
async function withBlindRetry(maxRetries) {
let physicalCalls = 0;
for (let i = 0; i <= maxRetries; i++) {
physicalCalls++;
try {
const res = await callDependency(i);
console.log(`Success after ${physicalCalls} physical call(s)`);
return res;
} catch (e) {
console.log(`Attempt ${i + 1} failed: ${e.message}`);
}
}
console.log(`Gave up after ${physicalCalls} physical calls`);
}
withBlindRetry(3);All lessons in this course
- Timeouts, Retries, and Exponential Backoff with Jitter
- The Circuit Breaker Pattern and Bulkhead Isolation
- Graceful Shutdown and In-Flight Request Draining
- Health Checks, Readiness Probes, and Load Shedding