Health Checks, Circuit Breakers, and Retry Logic

In distributed systems, components fail continuously — instances crash, network partitions occur, downstream services become overloaded. Without automated failure detection, traffic continues flowing to failed components, causing cascading failures. AWS provides multiple layers of health checking: ELB health checks detect unhealthy instances, Route 53 health checks detect unhealthy endpoints, and Auto Scaling replaces failed instances. Application-level patterns like circuit breakers and retries complete the resilience picture.

Elastic Load Balancer health checks periodically send requests to registered targets to determine if they are healthy. You configure the health check path (e.g., /health), protocol, port, interval (default 30 seconds), and healthy/unhealthy threshold (number of consecutive successes/failures). When a target fails health checks, the ELB stops routing traffic to it. The target is re-evaluated continuously and added back once it passes the healthy threshold.

Route 53 health checks monitor endpoints from multiple locations globally and work in tandem with DNS failover routing. Three types exist: Endpoint checks directly poll your application URL. Calculated checks combine multiple child health checks with AND/OR logic (useful for complex monitoring). CloudWatch alarm checks delegate health determination to CloudWatch — useful when you cannot expose a public health endpoint or need metric-based health decisions.

Auto Scaling Groups can use two types of health checks: EC2 health checks detect instance failures at the hypervisor level (instance status check failure). ELB health checks are more application-aware — an instance might be running but serving errors, and ELB health checks catch this. You can configure ASG to use ELB health checks so that application-level failures also trigger instance replacement, not just underlying EC2 failures.

A circuit breaker is an application-level pattern that prevents cascading failures by monitoring calls to a downstream service and temporarily stopping calls when failures exceed a threshold. The circuit has three states: Closed (normal operation), Open (failures exceeded threshold, calls are blocked immediately), and Half-Open (after a timeout, a small number of test calls are allowed to check if the service has recovered). AWS App Mesh and application SDKs like Resilience4j implement this pattern.

AWS App Mesh is a service mesh that implements circuit breaking, retries, and timeout policies at the infrastructure level without code changes. You define circuit breaker policies in the virtual node or virtual router configuration. When an upstream service becomes unhealthy, App Mesh's Envoy proxy automatically opens the circuit, returning errors immediately rather than waiting for timeouts. This is particularly valuable in microservices architectures running on ECS or EKS.

Retry logic automatically reattempts failed operations, but naive retry logic (immediate retry in a tight loop) can worsen overload situations. Exponential backoff increases the wait time between retries exponentially: 1s, 2s, 4s, 8s... This reduces load on a struggling service and gives it time to recover. Jitter (randomising retry intervals) prevents the thundering herd problem where all clients retry simultaneously after a brief outage. The AWS SDK implements exponential backoff with jitter automatically.

Retries are only safe if operations are idempotent — performing the same operation multiple times produces the same result. For example, creating an S3 object with the same key is idempotent (same result). But placing an order twice creates two orders — not idempotent. Design APIs to be idempotent using client-supplied idempotency keys: the server stores the result of the first request and returns the same result for subsequent requests with the same key. DynamoDB, SQS, and API Gateway support idempotency key patterns.

Without explicit timeouts, a slow downstream service causes threads to block indefinitely, exhausting the connection pool and causing cascading failures. Set timeouts at every layer: connection timeout (time to establish TCP connection), read timeout (time to receive a response), and overall request timeout. In AWS, configure ELB idle timeout (default 60s), Lambda execution timeout (max 15 min), and API Gateway integration timeout (max 29s). Timeouts trigger your retry or circuit-breaker logic.

When message processing fails repeatedly, a Dead Letter Queue (DLQ) captures messages that could not be processed after the maximum number of receive attempts. Configure DLQs on SQS queues and Lambda event source mappings to prevent bad messages from blocking your queue indefinitely. Messages in the DLQ can be inspected, replayed after fixing the bug, or archived. DLQs are a critical component of resilient event-driven architectures.

Effective health checks and circuit breakers require monitoring to understand failure patterns. CloudWatch is the observability layer: create alarms on ELB UnHealthyHostCount (instances failing health checks), Lambda Errors rate, SQS NumberOfMessagesSentToDLQ (messages hitting DLQ), and Target group RequestCountPerTarget. Set up SNS notifications so your on-call team is alerted immediately when automated health checks detect degradation.

Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.

In this lesson you learned: ELB and Route 53 health checks automate failure detection at the infrastructure level, circuit breakers prevent cascading failures by stopping calls to unhealthy services, and exponential backoff with jitter makes retries safe under load. Dead Letter Queues capture failed messages for inspection. Next up we explore RTO, RPO, and disaster recovery tiers.