DynamoDB Streams to Lambda
Lambda polls a DynamoDB stream and invokes a function with batches of item changes. What each side expects — stream view type, event source mapping settings (starting position, batch size, retries, bisect, partial batch responses, on-failure destination), ordering and duplicates — with a typed handler, a known-good mapping and the known-bad ones.
markdownpatternv4CWCV-R0V3stage: buildtrust: officialservice: dynamodbservice: lambdainterface: cliinterface: sdkDynamoDB Streams Lambdaevent source mapping DynamoDBTRIM_HORIZONLATEST misses eventsReportBatchItemFailuresBisectBatchOnFunctionErrorMaximumRetryAttempts infiniteMaximumRecordAgeInSecondson-failure destinationParallelizationFactorstream orderingpoison recordDynamoDBStreamEvent
Lambda, not DynamoDB, does the work: an event source mapping polls each stream shard about four times a second and invokes the function with a batch. The defaults will block a shard on one bad record for the full 24-hour retention, because retries are infinite. Set MaximumRetryAttempts, BisectBatchOnFunctionError and ReportBatchItemFailures, plus an on-failure destination.
What crosses the handoff
| Side | Provides / expects |
|---|---|
| DynamoDB | A stream on the table (choose a view type: new image, old image, both, or keys only; UNVERIFIED names). Records retained 24 hours. Ordering is guaranteed per item |
| Event source mapping | StartingPosition (required): TRIM_HORIZON or LATEST. Batch size 100 by default, up to 10,000. Batching window up to 5 minutes. Payload up to 6 MB. Parallelization factor 1–10 concurrent batches per shard |
| Function | Execution role that can read the stream (dynamodb:GetRecords, GetShardIterator, DescribeStream, ListStreams; UNVERIFIED list, not on the fetched pages). An idempotent handler, because delivery is at least once |
Known-good mapping
aws lambda create-event-source-mapping --function-name <fn> \
--event-source-arn <table stream ARN> --starting-position TRIM_HORIZON \
--batch-size 100 --maximum-batching-window-in-seconds 1 \
--maximum-retry-attempts 5 --maximum-record-age-in-seconds 3600 \
--bisect-batch-on-function-error --function-response-types ReportBatchItemFailures \
--destination-config '{"OnFailure":{"Destination":"<SQS queue or SNS topic ARN>"}}'
(Flag spellings follow the documented parameter names: BatchSize, MaximumBatchingWindowInSeconds, MaximumRetryAttempts, MaximumRecordAgeInSeconds, BisectBatchOnFunctionError, FunctionResponseTypes, DestinationConfig. The CLI forms are UNVERIFIED.)
import type { DynamoDBStreamEvent, DynamoDBBatchResponse } from "aws-lambda";
export const handler = async (event: DynamoDBStreamEvent): Promise<DynamoDBBatchResponse> => {
const batchItemFailures: { itemIdentifier: string }[] = [];
for (const r of event.Records) {
try {
if (r.eventName === "REMOVE") continue;
const newImage = r.dynamodb?.NewImage; // attribute-value format: { status: { S: "PAID" } }
if (newImage?.status?.S === "PAID") {
// idempotent side effect keyed on the item, e.g. a conditional write elsewhere
}
} catch {
batchItemFailures.push({ itemIdentifier: r.dynamodb!.SequenceNumber! });
}
}
return { batchItemFailures }; // requires FunctionResponseTypes: ["ReportBatchItemFailures"]
};
TTL expiries arrive as REMOVE records too, marked as service deletions rather than user deletes (reference/dynamodb/ttl-pitr-and-global-tables.md). Skip or handle them on purpose.
- Images arrive in attribute-value format (
{ S: "…" }), not the document client's plain values. - Return the failing record's
SequenceNumber. Lambda retries from there, not the whole batch. - On-failure destination: an SQS queue, SNS topic, S3 bucket or Kafka topic. The role needs
sqs:SendMessage,sns:Publish,s3:PutObject+s3:ListBucket, orkafka-cluster:WriteData. - SQS and SNS receive metadata only:
requestContext.condition(for exampleRetryAttemptsExhausted),responseContext.functionError, andDDBStreamBatchInfowithshardId,startSequenceNumber,endSequenceNumberandstreamArn. Read the records back from the stream before they expire. - S3 also receives the full invocation record in a
payloadfield, underaws/lambda/<ESM-UUID>/<shardID>/YYYY/MM/DD/…. - With default settings, one bad record can block its shard for up to a day, so set a retry count and a maximum record age.
Known-bad
| Configuration | Symptom |
|---|---|
Defaults: MaximumRetryAttempts -1, no bisect, no partial responses | One poison record blocks its shard and is retried until it expires (24 h). Later changes to items in that shard queue behind it |
StartingPosition: LATEST on a new mapping | Events written while the mapping is being created are missed: creation is eventually consistent. Use TRIM_HORIZON |
| Handler not idempotent | Duplicates on retries (at-least-once) |
| Handler returns success but swallowed errors | Changes are silently lost |
| More than two functions reading one shard (one for global tables) | Throttled reads (stream readers per shard limit) |
| Function writes back to the same table, and that write streams again | A feedback loop. Filter events (FilterCriteria) or write a marker the handler skips |
| Lambda extensions doing slow work after each batch | Lambda doesn't wait for them before the next batch, which can throttle the function |
Using the document client's types for NewImage | Type errors or undefined: unmarshall attribute values first (@aws-sdk/util-dynamodb; UNVERIFIED package name in the collected sources) |
Which side is wrong?
- Mapping state not
Enabled, orLastProcessingResultshows a permission error → the execution role (aws lambda get-event-source-mapping --uuid <id>). - Mapping enabled, iterator age growing → the function is failing or too slow. Check its errors and duration, and raise the parallelization factor.
- Some changes never processed →
LATESTstart, swallowed errors, or records expired after 24 h of retries. Check the on-failure destination.