0Pricing
NestJS Enterprise Backend APIs · 课时

使用断路器应对下游故障

集成类似 opossum 的断路器，保护服务免受级联故障影响。

使用断路器应对下游故障 是 CoddyKit 上的免费 NestJS Enterprise Backend APIs 课时。 这是第 2 节课，共 4 节。 你可以在下方免费阅读本课时的完整内容 — 然后在浏览器中使用内置代码编辑器和全天候 AI 导师进行实践。 这是 NestJS Enterprise Backend APIs 学习路径的一部分，你的进度在网页和 CoddyKit 应用中同步。 NestJS Enterprise Backend APIs 课程共包含 4 节课。

本课时的部分内容尚未翻译，以英文显示。

Why Cascading Failures Happen

In an enterprise NestJS backend, your service rarely lives alone. It calls a payment gateway, an auth provider, a search cluster, other microservices. When one downstream dependency slows down, every request waiting on it holds a connection, a thread, and a chunk of the event loop.

  • A slow dependency exhausts your HTTP connection pool.
  • Pending requests pile up; latency climbs everywhere.
  • Your service becomes unhealthy and its own callers start failing.

This domino effect is a cascading failure. A circuit breaker is the pattern that stops the dominoes from falling.

The Circuit Breaker State Machine

A circuit breaker wraps a risky call and tracks its health through three states:

  • CLOSED — calls flow through normally. Failures are counted.
  • OPEN — too many failures occurred; calls are rejected instantly without touching the dependency. This gives the downstream time to recover.
  • HALF_OPEN — after a cooldown, a few trial calls are allowed. If they succeed, the breaker closes; if they fail, it opens again.

The key insight: when the breaker is OPEN, you fail fast instead of waiting on a timeout for every request.

A Minimal Breaker From Scratch

Before reaching for a library, it helps to understand the mechanics. Here is a tiny, self-contained breaker in TypeScript that flips between CLOSED and OPEN based on consecutive failures and a reset timeout.

This runs standalone so you can watch the state transitions.

type State = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

class MiniBreaker {
  private state: State = 'CLOSED';
  private failures = 0;
  private openedAt = 0;
  constructor(private threshold = 3, private resetMs = 1000) {}

  async call<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === 'OPEN') {
      if (Date.now() - this.openedAt >= this.resetMs) this.state = 'HALF_OPEN';
      else throw new Error('Circuit OPEN: failing fast');
    }
    try {
      const result = await fn();
      this.failures = 0;
      this.state = 'CLOSED';
      return result;
    } catch (e) {
      this.failures++;
      if (this.failures >= this.threshold) {
        this.state = 'OPEN';
        this.openedAt = Date.now();
      }
      throw e;
    }
  }
  get current() { return this.state; }
}

async function main() {
  const breaker = new MiniBreaker(2, 500);
  const flaky = () => Promise.reject(new Error('downstream down'));
  for (let i = 0; i < 4; i++) {
    try { await breaker.call(flaky); }
    catch (e) { console.log(`call ${i}: ${(e as Error).message} [state=${breaker.current}]`); }
  }
}
main();

Enter opossum

Hand-rolled breakers miss the hard parts: rolling-window statistics, percentage-based thresholds, half-open trial limits, and metrics. opossum is the de-facto Node.js circuit breaker and integrates cleanly into NestJS providers.

  • timeout — how long before a call is considered failed.
  • errorThresholdPercentage — % of failures in the window that trips the breaker.
  • resetTimeout — how long OPEN lasts before a HALF_OPEN trial.
  • rollingCountTimeout — the size of the statistical window.

Install it with npm i opossum and npm i -D @types/opossum.

import CircuitBreaker from 'opossum';

const options: CircuitBreaker.Options = {
  timeout: 3000,                 // a call slower than 3s counts as a failure
  errorThresholdPercentage: 50,  // trip when >=50% of calls fail
  resetTimeout: 10000,           // stay OPEN for 10s, then try HALF_OPEN
  rollingCountTimeout: 10000,    // 10s statistical window
  rollingCountBuckets: 10,       // split the window into 10 buckets
};

// The action is the function we want to protect
async function fetchUser(id: string): Promise<{ id: string }> {
  // ... real HTTP call to a downstream user service ...
  return { id };
}

export const userBreaker = new CircuitBreaker(fetchUser, options);

Wrapping a Downstream Call in a NestJS Provider

In NestJS, the breaker belongs in a provider that owns one logical dependency. Build the CircuitBreaker once in the constructor (or a factory) so the rolling statistics persist across requests — never create a new breaker per request, or it can never learn the dependency's health.

Expose a method that delegates to breaker.fire(...).

import { Injectable } from '@nestjs/common';
import { HttpService } from '@nestjs/axios';
import { firstValueFrom } from 'rxjs';
import CircuitBreaker from 'opossum';

interface PricingDto { sku: string; cents: number; }

@Injectable()
export class PricingClient {
  private readonly breaker: CircuitBreaker<[string], PricingDto>;

  constructor(private readonly http: HttpService) {
    this.breaker = new CircuitBreaker(
      (sku: string) => this.requestPrice(sku),
      { timeout: 2000, errorThresholdPercentage: 50, resetTimeout: 15000 },
    );
  }

  private async requestPrice(sku: string): Promise<PricingDto> {
    const res = await firstValueFrom(
      this.http.get<PricingDto>(`https://pricing.internal/skus/${sku}`),
    );
    return res.data;
  }

  getPrice(sku: string): Promise<PricingDto> {
    return this.breaker.fire(sku);
  }
}

Fallbacks: Degrade Gracefully

Failing fast is good, but returning a hard error to the user is often worse than returning something reasonable. opossum's fallback() runs whenever the action rejects or the breaker is OPEN.

  • Serve a cached or last-known-good value.
  • Return a safe default (for example, an empty recommendations list).
  • Queue the work for later instead of dropping it.

The fallback receives the same arguments plus the triggering error, so you can branch on it.

import CircuitBreaker from 'opossum';

type Recommendation = { id: string };

function buildRecommendationBreaker(
  action: (userId: string) => Promise<Recommendation[]>,
) {
  const breaker = new CircuitBreaker(action, {
    timeout: 1500,
    errorThresholdPercentage: 40,
    resetTimeout: 20000,
  });

  // When OPEN or the action fails, return a safe empty list
  breaker.fallback((_userId: string, err?: Error) => {
    if (err) console.warn('recommendations degraded:', err.message);
    return [] as Recommendation[];
  });

  return breaker;
}

Listening to Breaker Events

A breaker that flips silently is an operational blind spot. opossum emits events for every meaningful transition. Wire these to your logger and metrics so on-call engineers see the picture in real time.

  • open / halfOpen / close — state transitions.
  • reject — a call rejected because the breaker was OPEN.
  • timeout — a call exceeded the configured timeout.
  • fallback — the fallback was invoked.
  • success / failure — outcome of each fired call.
import { Logger } from '@nestjs/common';
import CircuitBreaker from 'opossum';

export function attachBreakerTelemetry(
  breaker: CircuitBreaker,
  name: string,
  logger = new Logger('CircuitBreaker'),
) {
  breaker.on('open', () => logger.error(`[${name}] OPEN - failing fast`));
  breaker.on('halfOpen', () => logger.warn(`[${name}] HALF_OPEN - probing`));
  breaker.on('close', () => logger.log(`[${name}] CLOSED - recovered`));
  breaker.on('reject', () => logger.warn(`[${name}] call rejected (OPEN)`));
  breaker.on('timeout', () => logger.warn(`[${name}] call timed out`));
  breaker.on('fallback', () => logger.warn(`[${name}] fallback served`));
}

Timeout Is Part of the Breaker

A common mistake is setting the breaker's timeout longer than the underlying HTTP client timeout. If your axios timeout is 30s but the breaker timeout is 3s, opossum gives up at 3s — good — but the socket may still be held open downstream.

Align them: set the breaker timeout slightly below the transport timeout, and make sure the transport actually aborts. The whole point is to bound how long any single call can pin a resource.

import { Module } from '@nestjs/common';
import { HttpModule } from '@nestjs/axios';

@Module({
  imports: [
    HttpModule.register({
      timeout: 2500,        // axios aborts the socket at 2.5s
      maxRedirects: 0,
    }),
  ],
})
export class DownstreamModule {}

// Breaker timeout (e.g. 2000ms) should sit just BELOW the axios timeout
// so opossum records the failure while the socket is still being released.

One Breaker Per Dependency, Not Per App

Bulkheading means isolating failures so a sick dependency cannot drown a healthy one. Give each downstream its own breaker instance with thresholds tuned to its SLA.

  • A flaky analytics service can trip without affecting payments.
  • A latency-sensitive auth call gets a tight timeout; a batch report call gets a loose one.
  • Per-dependency metrics make dashboards readable.

Do not share a single global breaker across unrelated calls — their statistics would contaminate each other.

import { Injectable } from '@nestjs/common';
import CircuitBreaker from 'opossum';

@Injectable()
export class BreakerRegistry {
  private readonly breakers = new Map<string, CircuitBreaker>();

  get<TArgs extends unknown[], TRet>(
    name: string,
    action: (...args: TArgs) => Promise<TRet>,
    options: CircuitBreaker.Options,
  ): CircuitBreaker<TArgs, TRet> {
    const existing = this.breakers.get(name);
    if (existing) return existing as CircuitBreaker<TArgs, TRet>;
    const breaker = new CircuitBreaker(action, options);
    this.breakers.set(name, breaker);
    return breaker;
  }
}

Exposing Breaker Health to Probes

Operationally you want breaker state visible in a health endpoint and scraped by Prometheus. opossum exposes breaker.stats (counts of success, failure, timeout, reject) and breaker.opened / breaker.halfOpen booleans.

A custom Terminus health indicator can report OPEN breakers as a degraded — not necessarily down — status, so orchestrators do not needlessly kill a pod that is correctly shedding load.

import { Injectable } from '@nestjs/common';
import { HealthIndicator, HealthIndicatorResult } from '@nestjs/terminus';
import CircuitBreaker from 'opossum';

@Injectable()
export class BreakerHealthIndicator extends HealthIndicator {
  check(name: string, breaker: CircuitBreaker): HealthIndicatorResult {
    const isUp = !breaker.opened;
    return this.getStatus(name, isUp, {
      state: breaker.opened ? 'open' : breaker.halfOpen ? 'half_open' : 'closed',
      failures: breaker.stats.failures,
      timeouts: breaker.stats.timeouts,
      rejects: breaker.stats.rejects,
    });
  }
}

Tuning Thresholds Sanely

Bad thresholds are worse than no breaker. Tune against real traffic:

  • errorThresholdPercentage too low (e.g. 10%) trips on normal jitter; too high (e.g. 90%) defeats the purpose. 40-60% is a common starting band.
  • resetTimeout too short hammers a recovering service; too long delays recovery. Start around 10-30s.
  • Account for low traffic: with only 2 calls in the window, one failure is 50%. opossum's volumeThreshold requires a minimum number of calls before the percentage can trip.
import CircuitBreaker from 'opossum';

const options: CircuitBreaker.Options = {
  timeout: 2000,
  errorThresholdPercentage: 50,
  resetTimeout: 15000,
  rollingCountTimeout: 10000,
  rollingCountBuckets: 10,
  volumeThreshold: 10, // need >=10 calls in the window before % can trip
  // errorFilter lets you NOT count expected errors (e.g. 404) as failures:
  errorFilter: (err: { statusCode?: number }) => err?.statusCode === 404,
};

export { options };

Quick Check: Choosing the Right Behavior

Test your understanding of how a circuit breaker behaves under load.

Recap

You now know how to keep a downstream failure from taking down your NestJS service:

  • The pattern: a breaker moves between CLOSED, OPEN, and HALF_OPEN; OPEN means fail fast instead of waiting on timeouts.
  • opossum: configure timeout, errorThresholdPercentage, resetTimeout, and volumeThreshold, then wrap the call with new CircuitBreaker(action, options) and invoke breaker.fire().
  • Build once: create the breaker in a provider/registry so rolling statistics persist — one breaker per dependency for bulkheading.
  • Degrade: register a fallback() for cached or safe-default responses.
  • Observe: hook the open/close/reject/timeout events into logs and metrics, and surface state in a health probe.

Combined with retries (with backoff) and timeouts, circuit breakers are a cornerstone of resilient, SLO-friendly distributed systems.

常见问题解答

「使用断路器应对下游故障」课时是免费的吗？

是的 — 「使用断路器应对下游故障」的完整文本可在网页上免费阅读。要进行交互式练习（内置代码编辑器和全天候 AI 导师）并解锁 NestJS Enterprise Backend APIs 课程的其余内容，请升级到 CoddyKit PRO。 NestJS Enterprise Backend APIs 课程共包含 4 节课。

「使用断路器应对下游故障」这节课中我会学到什么？

集成类似 opossum 的断路器，保护服务免受级联故障影响。 你通过在浏览器中直接运行的动手代码来练习 NestJS Enterprise Backend APIs，全天候 AI 导师会在你学习这节课的过程中回答你的问题。

学习 NestJS Enterprise Backend APIs 需要有经验吗？

无需任何先前经验。CoddyKit 上的 NestJS Enterprise Backend APIs 课程适合初学者到高级学习者，你可以从这里开始或从头开始，按照自己的节奏学习。 这是第 2 节课，共 4 节。

「使用断路器应对下游故障」课时需要多长时间？

大多数 CoddyKit 课程大约需要 5–10 分钟。每节课都很精短且互动，所以你能稳步进步，并在网页和应用中从离开的地方继续。

我能在这节 NestJS Enterprise Backend APIs 课中编写并运行代码吗？

能。每节 NestJS Enterprise Backend APIs 课都包含内置代码编辑器，你可以在浏览器中直接编写并运行真实代码，并获得即时 AI 反馈 — 无需本地设置。

此课程中的所有课时

  1. 使用拦截器实现超时、重试与舱壁隔离
  2. 使用断路器应对下游故障
  3. 使用 OpenTelemetry 实现分布式追踪
  4. 定义 SLO 与错误预算
← 返回 NestJS Enterprise Backend APIs