Visibility Timeout, DLQ, and Long Polling

When a consumer receives a message from SQS, the message is hidden from all other consumers for a period called the visibility timeout. During this window, the consumer processes the message and then deletes it. If the consumer crashes or doesn't finish in time, the visibility timeout expires and the message becomes visible again, allowing another consumer to pick it up. This is the core mechanism behind SQS's at-least-once delivery guarantee.

The default visibility timeout is 30 seconds. It can be set from 0 seconds to 12 hours. Set it to comfortably exceed your maximum expected processing time—if processing takes up to 2 minutes, set the timeout to at least 3-4 minutes. You can also change the timeout per-receipt using change-message-visibility, which is useful when a consumer detects it needs more time to finish processing a specific message.

Setting the timeout too short causes messages to reappear before the consumer finishes, leading to duplicate processing. Setting it too long delays another consumer from picking up a message if the original consumer fails silently (e.g., EC2 instance crash without graceful cleanup). The optimal timeout is: slightly longer than your 99th-percentile processing time but short enough to recover quickly from consumer failures. Monitor the ApproximateNumberOfMessagesNotVisible CloudWatch metric to spot timeout issues.

A Dead-Letter Queue (DLQ) is a separate SQS queue where messages are sent after they fail processing a configurable number of times (the maxReceiveCount). When a message's receive count exceeds maxReceiveCount, SQS moves it to the DLQ automatically. DLQs prevent poison pill messages (messages that always fail) from blocking the queue indefinitely. Messages in the DLQ can be inspected, debugged, and replayed after fixing the processing bug.

A DLQ is just a regular SQS queue (Standard for Standard source, FIFO for FIFO source). You configure the Redrive Policy on the source queue to specify which queue is the DLQ and the maxReceiveCount threshold. Ensure the DLQ's retention period is longer than the source queue's retention period—messages arrive at the DLQ late, and you need time to investigate before they expire.

Set a CloudWatch alarm on the DLQ's ApproximateNumberOfMessagesVisible metric. Any message arriving in the DLQ indicates a processing failure that needs attention. Configure the alarm to trigger an SNS notification that pages your on-call engineer immediately. Treat every DLQ message as a bug that needs investigation—DLQ messages should not accumulate silently. After fixing the bug, use DLQ Redrive to send messages back to the source queue for reprocessing.

Once you have fixed the bug that caused processing failures, use SQS DLQ Redrive to move messages from the DLQ back to the source queue for reprocessing. The console provides a built-in redrive capability. You can filter by message attributes to replay only specific messages. Alternatively, write a Lambda function to poll the DLQ and forward messages back to the source queue if you need custom filtering logic.

By default, SQS uses short polling: a receive-message call samples a random subset of servers and returns immediately—even if no messages are available. This results in many empty responses and wastes API calls. Long polling waits up to 20 seconds for a message to arrive before returning an empty response. Long polling reduces costs significantly (fewer API calls) and decreases latency (message is received as soon as it arrives, up to 20s earlier than the next poll cycle).

Enable long polling at the queue level (applies to all receive calls) or per-request. Queue-level configuration with ReceiveMessageWaitTimeSeconds of 20 is the recommended setting for most applications. When Lambda uses SQS as an event source, it uses long polling automatically. For EC2-based consumers, set WaitTimeSeconds in the receive-message call.

SQS messages can carry message attributes—metadata key-value pairs separate from the message body. Attributes have a type (String, Number, Binary) and a value. When SQS is subscribed to an SNS topic, you can use SNS subscription filter policies applied to message attributes to route only relevant messages to each queue. Without filtering, every SQS subscriber receives every SNS publication regardless of content.

A Delay Queue makes every new message invisible for a delay period (0 to 15 minutes) after it is sent. This is useful for workflows where a consumer should not process a message immediately—for example, waiting for a dependent process to complete first. You can also set a per-message delay using DelaySeconds in the send call, which overrides the queue-level delay. Note: delay queues are not available for FIFO queues.

Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.

In this lesson you learned: Visibility Timeout hides a message from other consumers during processing, enabling at-least-once delivery with automatic re-delivery if the consumer fails, Dead-Letter Queues capture repeatedly failing messages for debugging and replay after the bug is fixed, and Long Polling (up to 20 seconds wait time) reduces API costs and latency compared to short polling. Next up we explore SNS topics and fan-out architecture.