SNS to SQS fan-out
Fan one SNS topic out to several SQS queues — the queue policy SNS needs (sns.amazonaws.com, sqs:SendMessage, aws:SourceArn), the customer managed KMS key when the queue or topic is encrypted, raw message delivery and its 10-attribute limit, per-subscription filter policies, SNS delivery retries to SQS (100,015 attempts over 23 days) and a subscription DLQ for permanent failures, cross-account confirmation, FIFO topics, and a TypeScript consumer for both envelope formats.
markdownpatternv4T0QY-VWPMstage: buildtrust: officialservice: snsservice: sqsinterface: iacinterface: sdkinterface: cliSNS SQS fan-outsubscribe SQS queue to SNS topicSQS queue policy sns.amazonaws.comaws:SourceArn topicSNS encrypted SQS KMS key policyaws/sqs SNS not deliveredRawMessageDeliverySNS envelope Type Notification Messageraw delivery 10 message attributes discardedSNS filter policy subscriptionSNS delivery retry SQSSNS subscription dead-letter queuecross-account SNS subscription confirmationSNS FIFO topic SQS FIFOCDK SqsSubscription
**Two policies decide whether a message arrives: the queue policy must let sns.amazonaws.com send from this topic, and, if the queue is encrypted, a customer managed key policy must let SNS use the key.** Missing either means nothing arrives and the publisher sees no error.
1. Queue policy (one statement per topic)
{
"Effect": "Allow",
"Principal": { "Service": "sns.amazonaws.com" },
"Action": "sqs:SendMessage",
"Resource": "arn:aws:sqs:<region>:<account-id>:<queue>",
"Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:sns:<region>:<account-id>:<topic>" } }
}
aws:SourceArn (or aws:SourceAccount when one queue listens to many topics) prevents the confused-deputy problem.
2. Encryption
- Encrypted queue: use SSE-SQS, or a customer managed key whose policy allows
sns.amazonaws.comkms:Decryptandkms:GenerateDataKey*.aws/sqswon't work, because its policy can't be changed (reference/sqs/queue-settings-and-encryption.md). - Encrypted topic: the SNS guide's tutorial uses one customer managed key for the topic and the queues, with this statement:
{
"Sid": "Allow Amazon SNS to use this key",
"Effect": "Allow",
"Principal": { "Service": "sns.amazonaws.com" },
"Action": ["kms:Decrypt", "kms:GenerateDataKey*"],
"Resource": "*"
}
Give the key an alias that doesn't start with aws/; that prefix is reserved for AWS managed keys.
3. Subscribe, filter, choose the format
aws sns subscribe --topic-arn <topic-arn> --protocol sqs --notification-endpoint <queue-arn> \
--attributes '{"RawMessageDelivery":"true","FilterPolicy":"{\"type\":[\"order_placed\"]}"}'
- Without raw delivery, each SQS message body is an SNS envelope:
Type: "Notification",MessageId,TopicArn,Subject,Message(your payload as a string),Timestamp,SignatureVersion,Signature,SigningCertURL,UnsubscribeURL. - With raw delivery (
RawMessageDelivery: true), the body is your message as published, and message attributes become SQS message attributes. More than 10 attributes means the message is discarded (a client-side error). Turn raw delivery off if you need more. - Filter policy: each subscription gets only matching messages. It filters on attributes by default, or on the body with
FilterPolicyScope: MessageBody. Changes take up to 15 minutes to apply (reference/sns/filter-policies.md). - Cross-account: when the subscriber isn't the queue owner, the subscription must be confirmed before messages flow.
4. When delivery fails
- SNS retries SQS (and Lambda) endpoints 100,015 times over 23 days: 3 immediate, 2 at 1 s, 10 with backoff from 1 to 20 s, then 100,000 at 20 s.
- Only 5XX and 429 are retried. Every other error is permanent and not retried.
- Once retries are exhausted, or on a permanent error, the message is discarded unless the subscription has a dead-letter queue. Give important subscriptions one: a
RedrivePolicysubscription attribute,{"deadLetterTargetArn": "<queue-arn>"}. That name and shape come from the CDK synth below; the retries page doesn't name it. The DLQ needs the samesns.amazonaws.comqueue policy. - Delivery semantics:
- Standard queues subscribed even to a FIFO topic get best-effort ordering and at-least-once delivery.
- Strict ordering and deduplication need an SNS FIFO topic plus SQS FIFO queues.
- Make consumers idempotent either way.
Known-good, known-bad, and which side is wrong
Known-good:
- The queue policy allows
sns.amazonaws.comsqs:SendMessagewithaws:SourceArn= the topic. - The queue uses SSE-SQS, or a customer managed key that grants SNS
kms:Decrypt+kms:GenerateDataKey*. - The subscription has a filter policy if needed, raw delivery (if consumers want the bare payload, with ≤10 attributes), and its own
RedrivePolicyDLQ. - The consumer is idempotent.
The CDK block below synthesizes to this configuration.
| Known-bad | Symptom |
|---|---|
Queue policy missing, or aws:SourceArn names another topic | Publish succeeds; the queue stays empty; the subscription DLQ (if any) receives the messages |
Queue encrypted with aws/sqs | Same symptom: SNS can't use a key whose policy it isn't in |
| Raw delivery with 11+ message attributes | Those messages never arrive (discarded as a client-side error) while others do |
| Consumer parses the body as the published JSON, but raw delivery is off | Consumer errors on every message: the body is the envelope, and the payload is the string in Message |
| Filter policy edited a minute ago | Old filtering still applies for up to 15 minutes |
Which side is wrong:
- Nothing arrives anywhere → the SNS-to-queue side: queue policy, KMS key, an unconfirmed cross-account subscription, or the filter. The subscription DLQ, if present, holds the rejected messages.
- Some arrive → filtering or the attribute count.
- Messages arrive but fail → the consumer side: envelope vs raw format. Then the consumer's own errors (
patterns/sqs-to-lambda.md).
Consumer that handles both formats
import type { SQSEvent, SQSBatchResponse } from "aws-lambda";
type Envelope = { Type?: string; Message?: string; TopicArn?: string };
function payloadOf(body: string): unknown {
const parsed = JSON.parse(body) as Envelope | unknown;
const env = parsed as Envelope;
// Non-raw delivery wraps the published message in an envelope; raw delivery doesn't.
if (env && env.Type === "Notification" && typeof env.Message === "string") {
try { return JSON.parse(env.Message); } catch { return env.Message; }
}
return parsed;
}
export const handler = async (event: SQSEvent): Promise<SQSBatchResponse> => {
const batchItemFailures: { itemIdentifier: string }[] = [];
for (const r of event.Records) {
try {
console.log(JSON.stringify(payloadOf(r.body)));
} catch {
batchItemFailures.push({ itemIdentifier: r.messageId });
}
}
return { batchItemFailures };
};
In the CDK
import { Stack } from "aws-cdk-lib";
import type { Construct } from "constructs";
import * as sns from "aws-cdk-lib/aws-sns";
import * as sqs from "aws-cdk-lib/aws-sqs";
import { SqsSubscription } from "aws-cdk-lib/aws-sns-subscriptions";
export function fanOut(scope: Construct) {
const stack = Stack.of(scope);
const topic = new sns.Topic(stack, "Orders");
const billing = new sqs.Queue(stack, "Billing", { encryption: sqs.QueueEncryption.SQS_MANAGED });
const subDlq = new sqs.Queue(stack, "BillingSubDlq");
topic.addSubscription(new SqsSubscription(billing, {
rawMessageDelivery: true,
filterPolicy: { type: sns.SubscriptionFilter.stringFilter({ allowlist: ["order_placed"] }) },
deadLetterQueue: subDlq,
}));
return { topic, billing };
}
Synthesizing it produced an AWS::SNS::Subscription with RawMessageDelivery, a FilterPolicy and a RedrivePolicy pointing at BillingSubDlq. It also produced AWS::SQS::QueuePolicy resources that allow sns.amazonaws.com sqs:SendMessage with aws:SourceArn set to the topic, on both the queue and the subscription DLQ. The CDK writes the queue policy for you.