Microservices Communication Patterns (Saga, Circuit Breaker) · Lesson

Circuit Breaker with Retry Logic

Understand the interplay between circuit breakers and retry mechanisms for optimal error handling and recovery.

Circuit Breaker with Retry Logic is a free Microservices Communication Patterns (Saga, Circuit Breaker) lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Microservices Communication Patterns (Saga, Circuit Breaker) learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Combining Circuit Breaker & Retry

In distributed systems, failures are inevitable. We've learned about the Retry Pattern for transient issues and the Circuit Breaker for persistent ones.

But how do these powerful patterns work together? Combining them effectively is key to building truly resilient microservices.

Recap: The Retry Pattern

The Retry Pattern automatically re-attempts an operation that has failed due to a temporary, transient error.

Use case: Network glitches, temporary service unavailability, database deadlocks.
Goal: Overcome momentary hiccups without user intervention.
Mechanism: Usually involves a delay between retries (e.g., exponential backoff).

Recap: The Circuit Breaker

A Circuit Breaker prevents an application from repeatedly invoking a service that is likely to fail. It "trips" the circuit to stop calls when too many errors occur.

Use case: Service is down, overloaded, or consistently returning errors.
Goal: Fail fast, prevent cascading failures, give the failing service time to recover.
States: Closed, Open, Half-Open.

Synergy: CB and Retry

Imagine a service experiencing a brief network blip. Retry can handle this gracefully. But what if the service is completely offline for an extended period?

Without a Circuit Breaker, retries would continuously hammer the unresponsive service, wasting resources and prolonging the problem. This is where their combined power shines!

Order of Operations

When combining these patterns, a critical design decision is: which one wraps the other?

Does the Retry Pattern wrap the Circuit Breaker, or does the Circuit Breaker wrap the Retry Pattern?

The order significantly impacts how your system responds to different types of failures.

Retry Wrapping Circuit Breaker

If the Retry Pattern wraps the Circuit Breaker:

Retry attempts the operation.
The Circuit Breaker is engaged.
If the CB opens, the first attempt fails, and retry might try again, hitting the already open CB.
This can lead to retries hitting a fast-failing CB, not allowing the CB to fully protect the system initially.

This setup is generally less effective.

Circuit Breaker Wrapping Retry

If the Circuit Breaker wraps the Retry Pattern:

The Circuit Breaker monitors the entire retry operation.
If the initial call fails, retry attempts again.
Only if all retries fail within the configured attempts, does the Circuit Breaker count it as a single failure.
If enough such "all-retry-failed" attempts occur, the CB opens.

This is the recommended approach.

CB Protecting Retry Logic

Here's a conceptual Java example showing how a Circuit Breaker would wrap an operation that includes retry logic. Notice the Circuit Breaker's decision to open or close is based on the final outcome of the retried call.

Try running this example:

import java.util.concurrent.atomic.AtomicInteger;

public class Main {
    // Simulate a dependency that sometimes fails
    private static AtomicInteger serviceCallCount = new AtomicInteger(0);

    public static boolean unreliableServiceCall() {
        System.out.println("  Attempting service call...");
        int currentCount = serviceCallCount.incrementAndGet();
        if (currentCount % 3 == 0) { // Fails every 3rd call
            System.out.println("  Service call FAILED temporarily.");
            return false;
        }
        System.out.println("  Service call SUCCESS.");
        return true;
    }

    public static boolean executeWithRetry() {
        int maxRetries = 2;
        long delayMillis = 100;
        for (int i = 0; i <= maxRetries; i++) {
            try {
                if (unreliableServiceCall()) {
                    return true; // Success after retry
                }
            } catch (Exception e) {
                // Log exception, continue retry
            }
            if (i < maxRetries) {
                System.out.println("  Retrying in " + delayMillis + "ms...");
                try { Thread.sleep(delayMillis); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
            }
        }
        return false; // All retries failed
    }

    // Conceptual Circuit Breaker logic for demonstration
    private static boolean circuitOpen = false;
    private static int failureCount = 0;
    private static final int FAILURE_THRESHOLD = 2; // Open after 2 consecutive failures
    private static final long RESET_TIMEOUT_MILLIS = 500; // Try to close after 0.5s
    private static long lastFailureTime = 0;

    public static boolean executeWithCircuitBreakerAndRetry() {
        if (circuitOpen) {
            if (System.currentTimeMillis() - lastFailureTime > RESET_TIMEOUT_MILLIS) {
                System.out.println("Circuit Breaker: Attempting HALF-OPEN state...");
                circuitOpen = false; // Move to half-open (for demo, just close)
                failureCount = 0; // Reset count
            } else {
                System.out.println("Circuit Breaker: OPEN! Failing fast.");
                return false; // Fail fast if open
            }
        }

        boolean success = executeWithRetry(); // Execute the retry logic

        if (!success) {
            failureCount++;
            lastFailureTime = System.currentTimeMillis();
            if (failureCount >= FAILURE_THRESHOLD) {
                circuitOpen = true;
                System.out.println("Circuit Breaker: OPENED due to repeated failures!");
            } else {
                System.out.println("Circuit Breaker: Failure detected, count=" + failureCount);
            }
        } else {
            failureCount = 0; // Reset failure count on success
            System.out.println("Circuit Breaker: Success, failure count reset.");
        }
        return success;
    }

    public static void main(String[] args) {
        System.out.println("--- Scenario: CB wrapping Retry ---");
        for (int i = 0; i < 7; i++) { // Simulate multiple requests
            System.out.println("\nRequest " + (i + 1) + ":");
            boolean overallSuccess = executeWithCircuitBreakerAndRetry();
            System.out.println("Overall result for Request " + (i + 1) + ": " + (overallSuccess ? "SUCCESS" : "FAILURE"));
            try { Thread.sleep(100); } catch (InterruptedException e) { Thread.currentThread().interrupt(); }
        }
    }
}

Why This Order is Best

Placing the Circuit Breaker around the Retry Pattern offers several advantages:

Efficient Failure Detection: The CB only opens after a series of genuinely failed operations (i.e., all retries failed), distinguishing transient issues from persistent outages.
Reduced Load: Once the CB is open, it prevents any further retry attempts, protecting the downstream service from being overwhelmed during a prolonged failure.
Faster Failures: When the service is truly down, the CB opens quickly, allowing your application to fail fast instead of waiting for all retries to exhaust.

Check Your Understanding

Consider a microservice that experiences intermittent network glitches (transient failures) and occasionally goes completely offline for maintenance (persistent failures).

You are implementing both the Retry Pattern and the Circuit Breaker Pattern to handle these scenarios. Which setup is generally recommended for optimal resilience?

Lesson Summary

We've explored the powerful synergy between the Circuit Breaker and Retry Patterns. While both enhance resilience, their combined effectiveness hinges on their interaction.

Remember, the best practice is to have the Circuit Breaker wrap the Retry Pattern. This allows retries to handle transient faults, while the Circuit Breaker steps in to protect against persistent failures, preventing cascading issues and improving overall system stability.

Keep building robust systems!

Frequently asked questions

Is the “Circuit Breaker with Retry Logic” lesson free?

Yes — the full text of “Circuit Breaker with Retry Logic” is free to read here on the web, and the Microservices Communication Patterns (Saga, Circuit Breaker) course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Microservices Communication Patterns (Saga, Circuit Breaker) course, upgrade to CoddyKit PRO.

What will I learn in “Circuit Breaker with Retry Logic”?

Understand the interplay between circuit breakers and retry mechanisms for optimal error handling and recovery. You practise Microservices Communication Patterns (Saga, Circuit Breaker) with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Microservices Communication Patterns (Saga, Circuit Breaker)?

No prior experience is required. Microservices Communication Patterns (Saga, Circuit Breaker) on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Circuit Breaker with Retry Logic” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Microservices Communication Patterns (Saga, Circuit Breaker) lesson?

Yes. Every Microservices Communication Patterns (Saga, Circuit Breaker) lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Microservices Communication Patterns (Saga, Circuit Breaker)