SQS Message Filtering and SNS + SQS Integration

In a fan-out architecture without filtering, every SQS subscriber receives every SNS message. If your topic publishes order events for 10 different product categories but a subscriber only processes electronics orders, it still receives and must discard food and clothing messages. This wastes compute, increases costs, and adds unnecessary load to consumers. SNS subscription filter policies solve this by having SNS itself route messages to only the appropriate subscribers.

A filter policy is a JSON object applied to an SQS or Lambda subscription. SNS evaluates the policy against each message's message attributes before delivering. If the message attributes match the filter policy, the message is delivered; if not, SNS skips that subscriber silently. Filter policies support string matching, numeric ranges, prefix matching, and the exists operator to check for attribute presence or absence.

By default, filter policies match against message attributes (metadata). As of 2023, SNS also supports payload-based (body) filtering by setting the filter policy scope to MessageBody. This allows JSON path-based filtering directly on the message body without requiring publishers to add attributes. Body filtering is more flexible but requires the message body to be valid JSON. Always confirm which scope matches your publisher's output format.

Filter policies support multiple matching operators beyond simple string equality:

• Numeric: {"numeric": ["=", 100]}, {"numeric": [">", 50, "<=", 200]}
• Prefix: {"prefix": "order-"} matches any string starting with that prefix
• Anything-but: {"anything-but": ["CANCELLED"]} matches any value except those listed
• Exists: {"exists": true} matches if the attribute is present; false if absent

For filtering to work, the publisher must include message attributes when publishing to SNS. Attributes are key-value pairs with a data type (String, Number, Binary). The publisher doesn't need to know which subscribers have which filter policies—it just enriches the message with attributes that describe the event. SNS handles routing automatically. This keeps publishers fully decoupled from subscriber-specific logic.

A sophisticated fan-out topology can have SNS routing to SQS queues at multiple specificity levels: one queue receives all orders (no filter) for auditing, another receives only HIGH_VALUE orders (amount >= 1000) for fraud review, and a third receives only ELECTRONICS orders for the electronics warehouse. Each SQS queue has its own Lambda consumer. This pattern scales each processing tier independently and adds new consumers without touching existing ones or the publisher.

SNS can deliver to SQS queues in a different AWS account. The SQS queue's resource-based policy must allow the SNS service principal to call sqs:SendMessage from the publishing account's SNS topic ARN. This enables centralised event publishing (one account publishes, multiple account teams subscribe) without sharing credentials. Cross-account fan-out is a common pattern in multi-account AWS Organizations setups.

When event volume spikes, direct SNS-to-Lambda invocations scale Lambda immediately, potentially overwhelming downstream databases or APIs. Adding SQS between SNS and Lambda creates a buffer: SNS delivers to SQS, and Lambda polls SQS at a controlled batch size. This lets Lambda process at a sustainable rate while SQS absorbs traffic spikes. The queue's depth acts as a backpressure mechanism—you can monitor it and alert when it grows beyond a threshold indicating consumer lag.

SNS → Lambda (direct): lowest latency, no buffering, Lambda scales immediately. Best for real-time alerting or urgent notifications where sub-second latency matters. SNS → SQS → Lambda: adds DLQ support, controlled throughput, retry with visibility timeout, and queue depth monitoring. Best for transaction processing, inventory updates, and any scenario where downstream limits must be respected. For exam questions, choose SQS buffering whenever durability and rate control are mentioned.

Use the SNS console filter policy editor to test whether a sample message would match your filter policy before deploying. You can also use the SNS Sandbox to simulate message delivery and verify routing. In code, validate filter policies by publishing test messages with known attributes and checking CloudWatch metrics for each subscription: the NumberOfMessagesFiltered metric shows how many messages were blocked by the filter, helping you tune policies without waiting for production events.

A complete SNS + SQS integration looks like this: (1) Application publishes an event to an SNS Standard topic with message attributes; (2) SNS evaluates each subscription's filter policy and delivers only matching messages to each SQS queue; (3) Lambda polls each SQS queue with a configured batch size and processes messages; (4) Failed messages reach the queue's DLQ after maxReceiveCount; (5) A CloudWatch alarm on DLQ depth alerts the team. This fully decoupled, resilient pattern is a model SAA-C03 architecture.

Test your understanding of AWS Solutions Architect (SAA-C03) concepts from this lesson.

In this lesson you learned: SNS filter policies route messages based on message attributes or body content at the SNS level, eliminating unnecessary processing by downstream consumers, the SNS→SQS→Lambda pattern adds durable buffering and rate control between fan-out broadcast and processing, and cross-account SQS delivery enables centralised pub/sub in multi-account environments using queue resource policies. Next up we explore Amazon API Gateway's REST, HTTP, and WebSocket APIs.