---
cuppy_hub: aws
cuppy_path: examples/healthcare-master-data/06-operations-scaling-and-performance.md
cuppy_section: docs
cuppy_doc: 1AYG-P6FX
cuppy_doc_uuid: 63177863-d693-45c2-8799-ca55c88465ba
cuppy_permalink: https://cuppy.io/aws/d/1AYG-P6FX
cuppy_version: 1
cuppy_updated: 2026-10-23T16:16:52.575Z
cuppy_canonical: https://cuppy.io/aws/docs/examples/healthcare-master-data/06-operations-scaling-and-performance
cuppy_mcp: https://cuppy.io/mcp
cuppy_body_sha256: 2b8859ec52851e2a706a6b44a9557a52fcb14eee82a85bdcbefb95c5db1eb7bf
# cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks.
---


# Operations, scaling and performance

## The throughput ceilings that are real

Scaling conversations go wrong when they stay abstract. These are the published numbers for AWS
Entity Resolution, read 2026-10-21, and they constrain the design more than compute does.

| Quota | Default | Adjustable |
| --- | --- | --- |
| Concurrent matching jobs | 1 per Region | No |
| Concurrent ID mapping jobs | 1 per Region | No |
| Records per rule-based matching workflow | 100,000,000 | Yes |
| Records per ML matching workflow | 600,000,000 in the larger Regions | Yes |
| Matching workflows | 10 per Region | Yes |
| `GetMatchId` rate | 50 per second | Yes |
| `GenerateMatchId` rate | 10 per second | Yes |
| Most control-plane API calls | 5 transactions per second | Documented as a rate limit |

The first row is the architectural constraint. One concurrent matching job per Region per account,
marked not adjustable. Four entity types therefore cannot match in parallel in one account and
Region. Three ways out, in order of preference:

1. **Serialise with Step Functions.** A state machine runs patient, then provider, then
   organisation, then location, each waiting on job completion. Simple, cheap, and the right answer
   when the total window fits the nightly batch.
2. **Separate accounts.** The quota is per account and per Region, so a second resolve account
   doubles effective concurrency. This is one of the reasons document 01 puts resolve in its own
   account.
3. **Separate Regions.** Possible, but it moves PHI across Regions and complicates the
   data-residency story. Prefer accounts.

Against the stated scale of 12 million patient records, the 100 million record ceiling for a
rule-based workflow is not binding, and the maximum of 20 data inputs comfortably covers six source
systems. The concurrency quota is the only one likely to bite.

## Batch and streaming, and which to use

Both paths exist in this architecture and they do different jobs.

| Property | Batch path | Synchronous path |
| --- | --- | --- |
| Trigger | Schedule, or automatic on new S3 data | API call from a registration screen |
| Mechanism | Entity Resolution matching workflow | `GenerateMatchId` |
| Scope | Whole dataset or incremental | One record per call |
| Techniques | Rule-based and ML | Rule-based only |
| Rate | Governed by the one-concurrent-job quota | 10 requests per second default |
| Output | Match groups in S3 | `matchGroups` in the response, also saved to S3 |

For the batch path, automatic incremental processing exists "for rule-based matching scenarios ... as
soon as new data is available in your Amazon S3 bucket", which keeps the master current without a
scheduler. Treat it carefully: an event-driven matcher that fires on every arriving file can run
more often than intended, and every run is billed per record.

For the synchronous path, `processingType` is the control. `CONSISTENT`, the default, "performs
immediate lookup and matching against all existing records, with results saved synchronously.
Provides highest accuracy but slower response time." `EVENTUAL` returns faster "with complete
matching results available later in S3". `EVENTUAL_NO_LOOKUP` "should only be used for records known
to be unique" — rarely true at patient registration, so not the default to reach for here. Advanced
matching workflows do not support the field.

A sensible division: streaming ingestion through Kinesis Data Streams or MSK so that ADT and
registration events land in the raw zone within seconds, with resolution following either
synchronously for the registration screen or on the next incremental run for everything else. Do not
promise sub-second master resolution for every event; promise sub-second *landing* and bounded
resolution latency.

## Idempotency and reprocessing

The pipeline must be safely re-runnable, because it will be re-run — after a bad load, a rule
change, or a recovery. Four rules make that true.

**Every stage is keyed, not appended.** The curated zone is keyed by source system and source record
id. The crosswalk is keyed by the same pair. `master_attribute` is keyed by master, attribute and
effective date. Re-running writes the same keys rather than duplicating rows.

**The resolve stage is a pure function.** Given the same match output, the same decision store and
the same survivorship rules, it must produce the same master. No clock reads, no random ordering, no
dependence on the previous run's result. This is what lets you reprocess twelve months to fix a rule
and compare the outcome.

**Steward decisions replay before survivorship.** Bulk processing "re-processes all data for any new
matches and updates to existing matches", and human verdicts are not part of that job's state.
Document 05 sets out the ordering; the operational point is that a rebuild is only safe because the
decision store is an input.

**Payloads never travel in orchestration state.** Standard workflows persist state input and output
in execution history, so a state machine passing patient demographics between states puts PHI in
that history and in CloudWatch. Pass S3 keys and crosswalk identifiers instead, never records. Where
a payload must transit, use the Express workflow's log-level controls and a KMS-encrypted log group,
and set retention. The same rule governs the publish contract: change events carry `master_id`,
entity type, version and change kind, and the consumer fetches the record under its own
authorisation — notify-then-pull, not push-the-data. That also makes a split event tractable, since
the consumer re-reads rather than reconciling two payloads.

**The source of truth for a rebuild is the raw zone.** Object Lock in governance mode protects raw
objects from deletion by most principals while allowing an explicit, permissioned bypass. Any master
value can be re-derived from those objects, which is both the recovery mechanism and the audit
answer to "where did this come from".

Deletion needs care in this design. A HIPAA retention schedule and an immutable raw zone are in
tension: Object Lock in compliance mode means an object "can't be overwritten or deleted by any
user, including the root user in your AWS account", and "the only way to delete an object under the
compliance mode before its retention date expires is to delete the associated AWS account". Choose
retention periods deliberately, use governance mode for the raw zone, and reserve compliance mode
for the audit archive where the retention requirement is genuinely fixed.

## Backup and disaster recovery

AWS Backup covers most of this architecture, and its own page warns against assuming: "Support for a
feature or service should not be assumed unless explicitly mentioned."

| Store | AWS Backup coverage per the cited page |
| --- | --- |
| Amazon S3 | Supported, with continuous backup and point-in-time restore, item-level restore |
| Amazon Aurora | Supported, continuous backup and PITR, cross-Region and cross-account copy |
| Amazon DynamoDB | Supported; far more features under AWS Backup advanced features |
| Amazon EFS, RDS, Redshift, Neptune, DocumentDB | Supported |
| AWS HealthLake | Not listed anywhere on the page |

Backup Vault Lock is listed among the features available for all supported resources, described as
"Write-once, read-many (WORM) with AWS Backup Vault Lock". Use it, plus a logically air-gapped
vault, plus cross-account copy into the security OU, so that a compromised credential in the data
accounts cannot destroy the recovery path.

### The HealthLake recovery gap

HealthLake is absent from AWS Backup's supported resources, so recovery for the FHIR store is a
build. What the service does give you is a native import "from an Amazon S3 bucket to an HealthLake
data store" and "Bulk data FHIR API export capabilities". The pattern that follows:

1. Scheduled Bulk Data export to an S3 bucket in a different account, versioned and Object Locked.
2. Export cost is a real line item — the pricing page shows export at "$0.19 per GB exported" — so
   choose a cadence deliberately rather than nightly by habit.
3. Keep the raw zone as the authoritative rebuild source. A HealthLake store can be reconstructed by
   re-importing from raw plus the resolved master, which is slower than a restore but is a complete
   recovery path and one you can rehearse.
4. Rehearse it. An untested rebuild is not a recovery plan.

Note also that resource matching evaluates identifiers "whenever a resource is written to your data
store – through a bulk import job or a REST API `create`, `update` or `delete`". A full re-import
therefore re-runs matching across the store, which affects both rebuild duration and the volume of
`Linkage` resources produced.

### Objectives by store

These are design targets for the stated scale, not service commitments.

| Store | RPO target | RTO target | Mechanism |
| --- | --- | --- | --- |
| S3 raw | Near zero | Minutes | Versioning, Object Lock, cross-Region replication |
| S3 curated | One run | Hours | Rebuild from raw |
| Aurora master | Minutes | Under an hour | Continuous backup and PITR, cross-Region copy |
| Match output | One run | Hours | Re-run the matching workflow |
| HealthLake | Export cadence | Hours to days | Export plus re-import; rehearsed |
| Crosswalk | Minutes | Under an hour | Aurora, plus the append-only journal in raw |

The crosswalk is the row to protect hardest. Losing the master's attribute values is recoverable
from raw; losing the mapping from source records to `master_id` makes every downstream system's
foreign keys meaningless.

Be precise about what the documentation supports here, because it is easy to overclaim in either
direction. Match IDs do persist for records already held in a live workflow's results:
`GenerateMatchId` "retrieves their Match IDs and associated rules" for existing records, and bulk
processing applies "updates to existing matches". What no page warrants is that a rebuilt workflow,
a changed rule set or a switch between rule-based and ML reproduces the same identifiers — which is
precisely the case a rebuild hits. So treat re-derivation as unavailable rather than impossible:
back the crosswalk up more often than anything else, and verify a restore of it specifically.

Two measures make it genuinely rebuildable rather than merely well-protected, and they are cheap
enough that there is no good reason to skip them. Write every crosswalk mutation as an append-only
journal record into the Object Lock raw zone, so the mapping derives from immutable storage like
every other value in the design. And derive `master_id` deterministically where the inputs allow —
a keyed hash of the earliest stable scoped identifier tuple — so a rebuild converges on the same
identifiers for the unambiguous majority, leaving only the steward-adjudicated minority to the
journal.

## Observability and run evidence

Each pipeline run should leave a record that answers an auditor's questions without a human
reconstructing them: which raw objects were consumed, the data quality score per source, the
matching workflow and job id, counts of match groups and of records processed, how many candidates
entered the review band, how many steward decisions were replayed, and how many master records
changed.

One availability note belongs here rather than in a design document, because it is an operational
commitment. The master sits on a synchronous clinical path, so a single instance is not an
acceptable posture: run an Aurora writer with a reader in a second Availability Zone and automatic
failover, and rehearse the failover. Define the degraded mode explicitly — if the master is
unreachable, registration falls back to `GenerateMatchId` alone and the golden-record update is
reconciled afterwards, rather than the screen blocking. Document 07 carries the cost of the reader.

Entity Resolution surfaces some of this itself — the console and API expose "the job status of
existing matching workflows and metrics, such as resource number, number of records processed, and
number of matches found". Glue Data Quality emits results to EventBridge and CloudWatch. Step
Functions gives per-execution history. Write the consolidated run record to a table in the master
and treat it as part of the deliverable, not as logging.

Two operational practices follow from the quotas rather than from good taste. Alarm on matching job
duration, because a serialised chain of four jobs has no slack if one slows down. And track the
adjustable quotas as configuration — records per workflow, workflow count, `GenerateMatchId` rate —
with a request raised before growth reaches them, since a quota increase is a support interaction
with a lead time, not a setting.


---
<!-- cuppy -->
You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 131 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/aws/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.
<!-- /cuppy -->