Building Resilient Serverless Systems
Design highly available and fault-tolerant serverless architectures by incorporating patterns like circuit breakers, retries, and idempotency across your functions.
Building Resilient Serverless Systems is a free Serverless AWS Lambda Development lesson on CoddyKit — lesson 2 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Serverless AWS Lambda Development learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.
Building Robust Serverless Systems
Welcome! In this lesson, we'll dive into designing highly resilient and fault-tolerant serverless applications. Even though AWS manages much of the infrastructure, your functions still need to handle failures gracefully.
We'll explore key architectural patterns to ensure your applications remain stable and performant, even when things go wrong.
The Reality of Distributed Systems
In a serverless world, your functions often interact with many other services: databases, APIs, message queues. These interactions happen over a network, and networks can be unreliable.
- Transient Failures: Brief network glitches or service slowdowns.
- Downstream Service Issues: A service your Lambda calls might be temporarily unavailable.
- Unexpected Data: Malformed input can cause your function to crash.
Designing for these "failures" is crucial for a stable system.
Lambda's Built-in Retry Logic
For certain invocation types, AWS Lambda automatically retries your function if it fails. This is a powerful built-in resilience mechanism for asynchronous invocations.
For example, if an SQS queue triggers your Lambda and your function errors, Lambda (or SQS) will retry the invocation a few times. This helps overcome transient issues without any code changes.
However, retries aren't a silver bullet; they can lead to duplicate processing if not handled carefully.
Making Operations Idempotent
When retries happen, your function might execute the same operation multiple times. This is where idempotency comes in.
An idempotent operation is one that can be applied multiple times without changing the result beyond the initial application.
- Example: Setting a value (
x = 5) is idempotent. - Non-Example: Incrementing a value (
x++) is NOT idempotent, as each retry would change the value.
For resilient systems, many operations should strive to be idempotent.
Keys to Idempotent Functions
To make your Lambda functions idempotent, you often need to track the state of a request. This typically involves:
- Generating a unique Idempotency Key for each request (e.g., from request ID, event source ID).
- Checking if this key has already been processed before performing the core logic.
- Storing the result or status of the operation associated with the key.
This ensures that even if a function is retried, the core side-effect only occurs once.
import hashlib
import json
# Imagine a database or cache for storing processed requests
# For simplicity, using a global dict here. A real app uses persistent storage.
processed_requests = {}
def is_idempotent(event_payload):
# Create a unique key from the event payload
# For a real app, use a proper hashing/unique ID strategy
event_hash = hashlib.md5(json.dumps(event_payload, sort_keys=True).encode('utf-8')).hexdigest()
if event_hash in processed_requests:
print(f"Request with hash {event_hash} already processed.")
return True
processed_requests[event_hash] = "processing" # Mark as processing
return False
def lambda_handler(event, context):
if is_idempotent(event):
return {
'statusCode': 200,
'body': json.dumps('Request already processed or is being processed.')
}
# Simulate actual work (e.g., writing to a database)
print(f"Processing new request: {event}")
# In a real scenario, update processed_requests[event_hash] = "completed"
# after successful processing and store the result in persistent storage.
return {
'statusCode': 200,
'body': json.dumps('Request processed successfully!')
}
Preventing Cascading Failures
The Circuit Breaker pattern is a powerful way to prevent a failing service from causing cascading failures throughout your application.
Imagine a call to an external API that starts failing. Continuously retrying it will just waste resources and slow down your function. A circuit breaker detects this and "opens" the circuit, stopping calls to the failing service temporarily.
This gives the failing service time to recover and prevents your application from getting bogged down.
How a Circuit Breaker Works
A circuit breaker typically has three states:
- Closed: Operations proceed as normal. If failures exceed a threshold, it transitions to Open.
- Open: All calls to the protected service immediately fail (or return a fallback). After a timeout, it transitions to Half-Open.
- Half-Open: A limited number of test calls are allowed through. If these succeed, it transitions back to Closed. If they fail, it returns to Open.
This intelligent behavior allows for self-healing.
Implementing a Simple Circuit Breaker
Implementing a full circuit breaker involves managing state (failures, success counts, last failure time). For serverless, this state might be stored in a shared cache (like ElastiCache) or a database.
While complex to implement from scratch in a simple Lambda, understanding the logic is key. Libraries exist for various languages to help, or you can leverage AWS services like Step Functions to orchestrate retry logic with delays.
import time
class CircuitBreaker:
def __init__(self, failure_threshold=3, reset_timeout=5):
self.state = "CLOSED"
self.failure_count = 0
self.last_failure_time = 0
self.failure_threshold = failure_threshold
self.reset_timeout = reset_timeout # seconds
def call(self, func, *args, **kwargs):
if self.state == "OPEN":
if time.time() - self.last_failure_time > self.reset_timeout:
self.state = "HALF-OPEN"
# In a real app, log this state change
else:
raise Exception("Circuit is open, service unavailable.")
try:
result = func(*args, **kwargs)
if self.state == "HALF-OPEN":
self.state = "CLOSED"
self.failure_count = 0
# In a real app, log this state change
return result
except Exception as e:
self.failure_count += 1
self.last_failure_time = time.time()
if self.failure_count >= self.failure_threshold:
self.state = "OPEN"
# In a real app, log this state change
raise e
Timeouts Prevent Hanging
Another crucial resilience pattern is using timeouts for external calls. If your Lambda function calls another service (e.g., a database, an HTTP API), that call could hang indefinitely if the service is unresponsive.
Configuring a timeout ensures your function doesn't wait forever, freeing up resources and allowing for retry logic to kick in faster. AWS Lambda itself has a configurable timeout, but you should also set timeouts within your code for specific external requests.
Resilient Design Challenge
Consider a Lambda function that processes incoming orders. If the function fails after successfully deducting payment but before updating the order status in a database, and then retries, what problem could arise if the payment deduction is NOT idempotent?
Summary: Building for Failure
We've covered essential patterns for building resilient serverless applications:
- Retries: Lambda's built-in mechanism for transient errors.
- Idempotency: Ensuring operations can be safely retried without unintended side-effects (e.g., duplicate charges).
- Circuit Breakers: Preventing cascading failures by intelligently stopping calls to failing services.
- Timeouts: Protecting against unresponsive external services.
By applying these principles, you can create serverless systems that gracefully handle inevitable failures.
Frequently asked questions
Is the “Building Resilient Serverless Systems” lesson free?
Yes — the full text of “Building Resilient Serverless Systems” is free to read here on the web, and the Serverless AWS Lambda Development course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Serverless AWS Lambda Development course, upgrade to CoddyKit PRO.
What will I learn in “Building Resilient Serverless Systems”?
Design highly available and fault-tolerant serverless architectures by incorporating patterns like circuit breakers, retries, and idempotency across your functions. You practise Serverless AWS Lambda Development with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.
Do I need any experience to start Serverless AWS Lambda Development?
No prior experience is required. Serverless AWS Lambda Development on CoddyKit is structured for beginners through advanced learners; this is — lesson 2 of 4, so you can start here or from the beginning and move at your own pace.
How long does the “Building Resilient Serverless Systems” lesson take?
Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.
Can I write and run code in this Serverless AWS Lambda Development lesson?
Yes. Every Serverless AWS Lambda Development lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.
All lessons in this course
- Canary and Blue/Green Deployments
- Building Resilient Serverless Systems
- Serverless Architectural Patterns
- Cost Optimization in Serverless Architectures