실패 처리를 위한 배달 못한 편지 큐(DLQ)

When building serverless applications, especially with asynchronous Lambda functions, what happens if an invocation fails repeatedly?

Without a proper mechanism, these failed events might simply be discarded, leading to data loss or unaddressed issues. This is where Dead Letter Queues (DLQs) come in.

First, let's quickly recap how asynchronous Lambda invocations work. When you invoke a Lambda function asynchronously (e.g., via S3, SNS, or direct API call with InvocationType: Event):

• Lambda places the event in an internal queue.
• It then attempts to invoke your function.
• If the function fails, Lambda automatically retries the invocation up to two times.

What happens if your Lambda function still fails after all automatic retries (initial attempt + two retries)?

By default, if no DLQ is configured, the event is simply discarded. This means you lose valuable information about what went wrong and the event data itself, making debugging and error recovery difficult.

A Dead Letter Queue (DLQ) is a destination for events that Lambda couldn't successfully process after exhausting all retry attempts.

Think of it as a 'parking lot' for problematic messages. Instead of disappearing, these failed events are sent to your chosen DLQ destination, allowing you to inspect, debug, and potentially re-process them later.

You can configure two types of AWS services as DLQ destinations for your Lambda functions:

• Amazon SQS (Simple Queue Service): A message queue. Failed events are sent to the SQS queue, where they await processing. This is a pull-based model.
• Amazon SNS (Simple Notification Service): A topic. Failed events are published to an SNS topic, which can then notify subscribers (e.g., email, other Lambda functions). This is a push-based model.

SQS is generally preferred for re-processing, while SNS is good for immediate notifications.

To use an SQS queue as a DLQ, you first need to create one. It's a standard SQS queue, but often named to indicate its purpose (e.g., my-lambda-dlq).

Here's how you might create a standard SQS queue using the AWS CLI:

Once your SQS queue is ready, you configure your Lambda function to use it as its DLQ. This involves updating the function's configuration.

You also need to ensure your Lambda's IAM execution role has permissions to send messages to the SQS queue (sqs:SendMessage).

Consider this Python Lambda function. It processes an event, but if the event contains "should_fail": true, it will raise an exception.

When invoked asynchronously, after retries, an event causing this failure would be sent to the configured DLQ.

Once events are in your DLQ, you can:

• Monitor: Use Amazon CloudWatch to track the number of messages in the DLQ.
• Inspect: View the content of the messages to understand the failure.
• Re-process: Move messages back to the original queue or trigger manual processing once the underlying issue is resolved.

This provides a crucial safety net for your asynchronous workflows.

You've learned about Dead Letter Queues and their importance. Let's test your understanding!

In this lesson, we explored Dead Letter Queues (DLQs) and their role in building resilient serverless applications. You learned:

• DLQs prevent data loss from failed asynchronous Lambda invocations.
• AWS SQS and SNS can serve as DLQ destinations.
• How to configure a Lambda function with a DLQ.
• The importance of monitoring and managing events in your DLQ.

DLQs are essential for robust error handling in event-driven architectures.