0Pricing
AI Engineering Academy · Leçon

Fournisseurs de secours et disjoncteurs

Créez une cascade de fournisseurs qui bascule automatiquement d’OpenAI vers Anthropic, puis vers un modèle local lorsque le fournisseur principal est lent ou indisponible, en utilisant le modèle du disjoncteur.

Fournisseurs de secours et disjoncteurs est une leçon AI Engineering Academy gratuite sur CoddyKit. Ceci est la leçon 3 sur 4. Tu peux lire la leçon complète ci-dessous gratuitement — puis la pratiquer en direct dans le navigateur avec un éditeur de code intégré et un tuteur IA 24/7. Elle fait partie du parcours d'apprentissage AI Engineering Academy, et ta progression se synchronise sur le web et l'application CoddyKit. Le cours AI Engineering Academy comprend 4 leçons au total.

Certaines parties de cette leçon n'ont pas encore été traduites et s'affichent en anglais.

Single Provider Risk

Relying on a single LLM provider creates a single point of failure. OpenAI has experienced outages that took minutes to hours to resolve. If your entire application depends on GPT-4o being available, any provider incident immediately translates to user-facing downtime. A fallback provider strategy maintains service continuity by routing to alternative providers when the primary fails.

Defining a Provider Cascade

A provider cascade is an ordered list of providers and models tried in sequence. When the primary fails or times out, the system automatically tries the next provider. A typical cascade might be: OpenAI GPT-4o → Anthropic Claude 3.5 Sonnet → a locally deployed Llama model. Each level is a fallback with the local model serving as the last resort that cannot go down.

from dataclasses import dataclass
from typing import Optional

@dataclass
class Provider:
    name: str
    base_url: Optional[str]
    api_key_env: str
    model: str
    priority: int  # lower = higher priority

CASCADE = [
    Provider('openai',    None,                              'OPENAI_API_KEY',    'gpt-4o',              1),
    Provider('anthropic', 'https://api.anthropic.com/v1',   'ANTHROPIC_API_KEY', 'claude-3-5-sonnet',   2),
    Provider('local',     'http://localhost:8000/v1',        'LOCAL_KEY',         'llama-3.1-8b-inst',   3),
]

Implementing the Fallback Loop

Implement the fallback loop as a simple try/except that iterates through the cascade. Catch transient errors (timeouts, 500s, 503s) and move to the next provider. Do not catch authentication errors (401) or invalid request errors (400) — these are programming mistakes that should surface immediately rather than fail over to another provider.

import openai
import os

TRANSIENT_ERRORS = (openai.APITimeoutError, openai.InternalServerError, openai.APIConnectionError)

async def call_with_fallback(messages: list, **kwargs) -> str:
    for provider in CASCADE:
        try:
            client = openai.AsyncOpenAI(
                api_key=os.environ[provider.api_key_env],
                base_url=provider.base_url
            )
            resp = await client.chat.completions.create(
                model=provider.model,
                messages=messages,
                timeout=10.0,
                **kwargs
            )
            return resp.choices[0].message.content
        except TRANSIENT_ERRORS as e:
            print(f'Provider {provider.name} failed: {e}, trying next...')
    raise RuntimeError('All providers failed')

What Is a Circuit Breaker?

A circuit breaker prevents a failing service from being hammered with requests during an outage. Named after electrical circuit breakers, it has three states: Closed (requests pass through normally), Open (requests are immediately rejected), and Half-Open (a single test request is allowed through to check if service has recovered). This protects both the downstream service and your own application during incidents.

# Circuit breaker state machine:
#
# CLOSED --> (failure_count >= threshold) --> OPEN
#    ^                                          |
#    |     (test_request succeeds)              | (timeout expires)
#    +------------ HALF_OPEN <-----------------+
#
# In OPEN state: immediately return fallback/error
# In HALF_OPEN: allow one request through to test recovery
# In CLOSED: normal operation, count failures

Implementing a Circuit Breaker

Here is a minimal circuit breaker implementation. Track the failure count and the time the circuit opened. When failure count exceeds the threshold, open the circuit. After a configurable reset timeout, allow one probe request. If the probe succeeds, close the circuit. If it fails, keep the circuit open and reset the timeout.

import time
from enum import Enum

class State(Enum):
    CLOSED = 'closed'
    OPEN = 'open'
    HALF_OPEN = 'half_open'

class CircuitBreaker:
    def __init__(self, failure_threshold=5, reset_timeout=60):
        self.state = State.CLOSED
        self.failure_count = 0
        self.failure_threshold = failure_threshold
        self.reset_timeout = reset_timeout
        self.opened_at = None

    def record_success(self):
        self.failure_count = 0
        self.state = State.CLOSED

    def record_failure(self):
        self.failure_count += 1
        if self.failure_count >= self.failure_threshold:
            self.state = State.OPEN
            self.opened_at = time.time()

    def can_attempt(self) -> bool:
        if self.state == State.CLOSED:
            return True
        if self.state == State.OPEN:
            if time.time() - self.opened_at > self.reset_timeout:
                self.state = State.HALF_OPEN
                return True  # allow one probe
            return False
        return True  # HALF_OPEN: allow probe

Integrating Circuit Breakers with Providers

Maintain one circuit breaker per provider. Before calling a provider, check if its circuit breaker allows the attempt. After each call, record success or failure. When a provider's circuit opens, the fallback loop naturally skips it and tries the next provider in the cascade, without waiting for a timeout on every single call.

breakers = {p.name: CircuitBreaker(failure_threshold=5, reset_timeout=60) for p in CASCADE}

async def call_with_circuit_breaker(messages: list) -> str:
    for provider in CASCADE:
        breaker = breakers[provider.name]
        if not breaker.can_attempt():
            continue  # skip this provider, circuit is open
        try:
            result = await call_provider(provider, messages)
            breaker.record_success()
            return result
        except TRANSIENT_ERRORS as e:
            breaker.record_failure()
            print(f'{provider.name} failed ({breaker.failure_count}/{breaker.failure_threshold})')
    raise RuntimeError('All providers exhausted')

Detecting Slow Calls as Failures

A provider that responds in 30 seconds is almost as bad as one that is fully down, from a user experience perspective. Configure an aggressive timeout per provider call and treat timeout exceptions as failures in the circuit breaker. A 10-second timeout means the fallback kicks in quickly enough that the user sees only a brief delay, not a hung screen.

async def call_provider(provider: Provider, messages: list) -> str:
    client = openai.AsyncOpenAI(
        api_key=os.environ[provider.api_key_env],
        base_url=provider.base_url
    )
    try:
        resp = await asyncio.wait_for(
            client.chat.completions.create(model=provider.model, messages=messages),
            timeout=10.0  # fail fast, let circuit breaker count it
        )
        return resp.choices[0].message.content
    except asyncio.TimeoutError:
        raise openai.APITimeoutError('Provider timed out')

Provider Health Dashboard

Expose a /health/providers endpoint that shows the current circuit breaker state for each provider, including failure count, state (closed/open/half-open), and time until reset. This makes it easy to see at a glance which providers are healthy during an incident and helps you decide whether to manually force a reset or wait for automatic recovery.

from fastapi import FastAPI

app = FastAPI()

@app.get('/health/providers')
def provider_health():
    return {
        name: {
            'state': cb.state.value,
            'failure_count': cb.failure_count,
            'seconds_until_reset': (
                max(0, cb.reset_timeout - (time.time() - cb.opened_at))
                if cb.state == State.OPEN else None
            )
        }
        for name, cb in breakers.items()
    }

Aligning Provider Outputs

Different providers have different response formats, safety filters, and capabilities. When switching from GPT-4o to Claude, the model might refuse certain requests that GPT-4o would answer. Maintain provider-specific prompt wrappers that adapt your prompts to each provider's conventions. Test each fallback provider independently to ensure it produces acceptable output for your use case.

def adapt_messages_for_provider(provider: Provider, messages: list) -> list:
    if provider.name == 'anthropic':
        # Claude prefers explicit task descriptions
        system = next((m['content'] for m in messages if m['role'] == 'system'), '')
        if 'JSON' not in system:
            messages = [{'role': 'system', 'content': system + ' Respond in JSON.'}] + [
                m for m in messages if m['role'] != 'system'
            ]
    return messages

Testing Fallback Behavior

Write a test that forces the primary provider to fail (by providing a bad API key or a mock that throws errors) and verifies the fallback kicks in and returns a valid response. Also test that the circuit breaker opens correctly after the configured number of failures and that it recovers after the reset timeout. Fallback logic that is never tested is unreliable in a real outage.

import pytest
from unittest.mock import AsyncMock, patch

@pytest.mark.asyncio
async def test_fallback_on_primary_timeout():
    # Primary provider times out
    with patch('your_module.call_provider', side_effect=[
        openai.APITimeoutError('Timeout'),  # primary fails
        'Claude response'                   # fallback succeeds
    ]):
        result = await call_with_circuit_breaker([{'role': 'user', 'content': 'Hello'}])
    assert result == 'Claude response'

Provider Cascade Cost Considerations

Fallback providers often have different pricing than your primary. Anthropic Claude may cost more or less than OpenAI GPT-4o depending on model tier. Track which provider served each request and compute cost attribution separately. If the fallback is consistently more expensive, investigate whether the primary is under-provisioned and whether upgrading to a higher rate-limit tier would be more cost-effective than frequent fallback usage.

# Approximate costs per 1M tokens (2026):
PROVIDER_COSTS = {
    'openai/gpt-4o':          {'input': 2.50, 'output': 10.00},
    'anthropic/claude-3.5-sonnet': {'input': 3.00, 'output': 15.00},
    'openai/gpt-4o-mini':     {'input': 0.15, 'output': 0.60},
    'local/llama-3.1-8b':     {'input': 0.00, 'output': 0.00},  # infra cost only
}

# If fallback adds $0.50/day and a Tier 2 upgrade costs $100/month:
# Tier 2 pays off if you use fallback > 200 requests/day

Quick Check

Test your understanding of circuit breakers and fallback providers.

Lesson Recap

In this lesson you learned: provider cascades define an ordered fallback sequence from primary to backup LLM providers, circuit breakers prevent hammering a failing provider by quickly short-circuiting after a threshold of failures, and per-provider timeouts ensure slow calls trigger fallback quickly rather than blocking users. Next up we set timeout budgets and implement graceful degradation.

Questions Fréquemment Posées

La leçon « Fournisseurs de secours et disjoncteurs » est-elle gratuite ?

Oui — le texte complet de « Fournisseurs de secours et disjoncteurs » est gratuit à lire ici sur le web. Pour la pratiquer de manière interactive (un éditeur de code intégré et un tuteur IA 24/7) et déverrouiller le reste du cours AI Engineering Academy, passe à CoddyKit PRO. Le cours AI Engineering Academy comprend 4 leçons au total.

Qu'est-ce que j'apprendrai dans « Fournisseurs de secours et disjoncteurs » ?

Créez une cascade de fournisseurs qui bascule automatiquement d’OpenAI vers Anthropic, puis vers un modèle local lorsque le fournisseur principal est lent ou indisponible, en utilisant le modèle du d… Tu pratiques AI Engineering Academy avec du code pratique que tu exécutes directement dans le navigateur, et un tuteur IA 24/7 répond à tes questions au fur et à mesure que tu avances dans la leçon.

Dois-je avoir de l'expérience pour commencer AI Engineering Academy ?

Aucune expérience préalable n'est requise. AI Engineering Academy sur CoddyKit est structuré pour les débutants jusqu'aux apprenants avancés, donc tu peux commencer ici ou depuis le début et avancer à ton rythme. Ceci est la leçon 3 sur 4.

Combien de temps prend la leçon « Fournisseurs de secours et disjoncteurs » ?

La plupart des leçons CoddyKit prennent environ 5–10 minutes. Chacune est courte et interactive, tu progresses régulièrement et tu repiques exactement où tu t'es arrêté sur le web et l'app.

Peux-tu écrire et exécuter du code dans cette leçon AI Engineering Academy ?

Oui. Chaque leçon AI Engineering Academy inclut un éditeur de code intégré, tu écris et exécutes du vrai code directement dans ton navigateur et tu reçois des retours IA instantanés — aucune configuration locale requise.

Toutes les leçons de ce cours

  1. Mesurer la latence des LLM : TTFT et TPOT
  2. Équilibrage de charge et stratégies multi-clés
  3. Fournisseurs de secours et disjoncteurs
  4. Budgets de délai et dégradation progressive
← Retour à AI Engineering Academy