--- cuppy_hub: aws cuppy_path: patterns/sns-to-sqs-fanout.md cuppy_section: docs cuppy_doc: T0QY-VWPM cuppy_doc_uuid: 781f4527-c553-442f-a8d9-6000f51008e8 cuppy_permalink: https://cuppy.io/aws/d/T0QY-VWPM cuppy_version: 4 cuppy_updated: 2026-10-23T16:16:52.575Z cuppy_canonical: https://cuppy.io/aws/docs/patterns/sns-to-sqs-fanout cuppy_mcp: https://cuppy.io/mcp cuppy_body_sha256: 7510962d1dd2339b17785691b7e6148eb96307cde47b70eae2740f4796d0b122 # cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks. --- # SNS to SQS fan-out **Two policies decide whether a message arrives: the queue policy must let `sns.amazonaws.com` send from *this* topic, and, if the queue is encrypted, a customer managed key policy must let SNS use the key.** Missing either means nothing arrives and the publisher sees no error. ## 1. Queue policy (one statement per topic) ```json { "Effect": "Allow", "Principal": { "Service": "sns.amazonaws.com" }, "Action": "sqs:SendMessage", "Resource": "arn:aws:sqs:::", "Condition": { "ArnEquals": { "aws:SourceArn": "arn:aws:sns:::" } } } ``` `aws:SourceArn` (or `aws:SourceAccount` when one queue listens to many topics) prevents the confused-deputy problem. ## 2. Encryption - **Encrypted queue:** use SSE-SQS, or a **customer managed** key whose policy allows `sns.amazonaws.com` `kms:Decrypt` and `kms:GenerateDataKey*`. `aws/sqs` won't work, because its policy can't be changed (`reference/sqs/queue-settings-and-encryption.md`). - **Encrypted topic:** the SNS guide's tutorial uses one customer managed key for the topic and the queues, with this statement: ```json { "Sid": "Allow Amazon SNS to use this key", "Effect": "Allow", "Principal": { "Service": "sns.amazonaws.com" }, "Action": ["kms:Decrypt", "kms:GenerateDataKey*"], "Resource": "*" } ``` Give the key an alias that doesn't start with `aws/`; that prefix is reserved for AWS managed keys. ## 3. Subscribe, filter, choose the format ``` aws sns subscribe --topic-arn --protocol sqs --notification-endpoint \ --attributes '{"RawMessageDelivery":"true","FilterPolicy":"{\"type\":[\"order_placed\"]}"}' ``` - **Without raw delivery,** each SQS message body is an SNS **envelope**: `Type: "Notification"`, `MessageId`, `TopicArn`, `Subject`, `Message` (your payload as a string), `Timestamp`, `SignatureVersion`, `Signature`, `SigningCertURL`, `UnsubscribeURL`. - **With raw delivery** (`RawMessageDelivery: true`), the body is your message as published, and message attributes become SQS message attributes. **More than 10 attributes means the message is discarded** (a client-side error). Turn raw delivery off if you need more. - **Filter policy:** each subscription gets only matching messages. It filters on attributes by default, or on the body with `FilterPolicyScope: MessageBody`. Changes take up to 15 minutes to apply (`reference/sns/filter-policies.md`). - **Cross-account:** when the subscriber isn't the queue owner, the subscription must be **confirmed** before messages flow. ## 4. When delivery fails - SNS retries SQS (and Lambda) endpoints **100,015 times over 23 days**: 3 immediate, 2 at 1 s, 10 with backoff from 1 to 20 s, then 100,000 at 20 s. - Only **5XX and 429** are retried. Every other error is permanent and not retried. - Once retries are exhausted, or on a permanent error, the message is **discarded unless the subscription has a dead-letter queue**. Give important subscriptions one: a `RedrivePolicy` subscription attribute, `{"deadLetterTargetArn": ""}`. That name and shape come from the CDK synth below; the retries page doesn't name it. The DLQ needs the same `sns.amazonaws.com` queue policy. - Delivery semantics: - Standard queues subscribed even to a **FIFO** topic get best-effort ordering and **at-least-once** delivery. - Strict ordering and deduplication need an SNS FIFO topic plus SQS FIFO queues. - Make consumers idempotent either way. ## Known-good, known-bad, and which side is wrong **Known-good:** - The queue policy allows `sns.amazonaws.com` `sqs:SendMessage` with `aws:SourceArn` = the topic. - The queue uses SSE-SQS, or a customer managed key that grants SNS `kms:Decrypt` + `kms:GenerateDataKey*`. - The subscription has a filter policy if needed, raw delivery (if consumers want the bare payload, with ≤10 attributes), and its own `RedrivePolicy` DLQ. - The consumer is idempotent. The CDK block below synthesizes to this configuration. | Known-bad | Symptom | |---|---| | Queue policy missing, or `aws:SourceArn` names another topic | `Publish` succeeds; the queue stays empty; the subscription DLQ (if any) receives the messages | | Queue encrypted with `aws/sqs` | Same symptom: SNS can't use a key whose policy it isn't in | | Raw delivery with 11+ message attributes | Those messages never arrive (discarded as a client-side error) while others do | | Consumer parses the body as the published JSON, but raw delivery is **off** | Consumer errors on every message: the body is the envelope, and the payload is the string in `Message` | | Filter policy edited a minute ago | Old filtering still applies for up to 15 minutes | **Which side is wrong:** - **Nothing arrives anywhere** → the SNS-to-queue side: queue policy, KMS key, an unconfirmed cross-account subscription, or the filter. The subscription DLQ, if present, holds the rejected messages. - **Some arrive** → filtering or the attribute count. - **Messages arrive but fail** → the consumer side: envelope vs raw format. Then the consumer's own errors (`patterns/sqs-to-lambda.md`). ## Consumer that handles both formats ```ts import type { SQSEvent, SQSBatchResponse } from "aws-lambda"; type Envelope = { Type?: string; Message?: string; TopicArn?: string }; function payloadOf(body: string): unknown { const parsed = JSON.parse(body) as Envelope | unknown; const env = parsed as Envelope; // Non-raw delivery wraps the published message in an envelope; raw delivery doesn't. if (env && env.Type === "Notification" && typeof env.Message === "string") { try { return JSON.parse(env.Message); } catch { return env.Message; } } return parsed; } export const handler = async (event: SQSEvent): Promise => { const batchItemFailures: { itemIdentifier: string }[] = []; for (const r of event.Records) { try { console.log(JSON.stringify(payloadOf(r.body))); } catch { batchItemFailures.push({ itemIdentifier: r.messageId }); } } return { batchItemFailures }; }; ``` ## In the CDK ```ts import { Stack } from "aws-cdk-lib"; import type { Construct } from "constructs"; import * as sns from "aws-cdk-lib/aws-sns"; import * as sqs from "aws-cdk-lib/aws-sqs"; import { SqsSubscription } from "aws-cdk-lib/aws-sns-subscriptions"; export function fanOut(scope: Construct) { const stack = Stack.of(scope); const topic = new sns.Topic(stack, "Orders"); const billing = new sqs.Queue(stack, "Billing", { encryption: sqs.QueueEncryption.SQS_MANAGED }); const subDlq = new sqs.Queue(stack, "BillingSubDlq"); topic.addSubscription(new SqsSubscription(billing, { rawMessageDelivery: true, filterPolicy: { type: sns.SubscriptionFilter.stringFilter({ allowlist: ["order_placed"] }) }, deadLetterQueue: subDlq, })); return { topic, billing }; } ``` Synthesizing it produced an `AWS::SNS::Subscription` with `RawMessageDelivery`, a `FilterPolicy` and a `RedrivePolicy` pointing at `BillingSubDlq`. It also produced `AWS::SQS::QueuePolicy` resources that allow `sns.amazonaws.com` `sqs:SendMessage` with `aws:SourceArn` set to the topic, on both the queue and the subscription DLQ. The CDK writes the queue policy for you. --- You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 131 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/aws/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.