--- cuppy_hub: aws cuppy_path: examples/healthcare-master-data/05-stewardship-and-data-quality.md cuppy_section: docs cuppy_doc: 97E8-77RA cuppy_doc_uuid: 69efbf74-2340-408b-9ea0-351af2fb1fe7 cuppy_permalink: https://cuppy.io/aws/d/97E8-77RA cuppy_version: 1 cuppy_updated: 2026-10-23T16:16:52.575Z cuppy_canonical: https://cuppy.io/aws/docs/examples/healthcare-master-data/05-stewardship-and-data-quality cuppy_mcp: https://cuppy.io/mcp cuppy_body_sha256: cae606660e538ab06947f0f88df13b9d6f8f18be75dcf4202f36b528febef047 # cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks. --- # Stewardship and data quality Data quality and stewardship are the two halves of governance. AWS supplies a capable managed service for the first and nothing for the second. This document covers what Glue Data Quality measures, precisely where it stops, and the shape of the steward application you must build. ## AWS Glue Data Quality AWS describes it as a service that "allows you to measure and monitor the quality of your data so that you can make good business decisions. Built on top of the open-source DeeQu [sic] framework, AWS Glue Data Quality provides a managed, serverless experience." Rules are written in DQDL, "a domain specific language that you use to define data quality rules", with "25+ out-of-the-box DQ rules to start from". The quality score is defined simply: "The percentage of data quality rules that pass (result in true) when you evaluate a ruleset." That gives a single trackable number per source per load, which is the metric a steward actually manages. ### The two entry points differ, and the difference decides your design "There are two entry points for AWS Glue Data Quality: the AWS Glue Data Catalog and AWS Glue ETL jobs." They are not equivalent, and the asymmetry is the single most practical thing to know: | Feature | Data Catalog entry point | ETL jobs entry point | | --- | --- | --- | | Data Quality rule recommendations | Supported | Not supported | | Author and run DQDL rules | Supported | Supported | | Identifying records that failed checks | Not supported | Supported | | Auto scaling | Not supported | Supported | | ML-based anomaly detection | Supported | Supported | | EventBridge and CloudWatch integration | Supported | Supported | | Incremental data quality | Pushdown predicates | Glue bookmarks | Read rows one and three together. Recommendations come from the catalog; the identity of failing rows comes from ETL. So the sensible pattern is to use the catalog entry point once, at design time, to have AWS propose a starting ruleset — "AWS Glue Data Quality quickly analyzes your data and creates data quality rules for you" — then move those rules into the ETL job, where you can quarantine the rows that failed. AWS's own framing supports this: the ETL entry point "lets you perform *proactive* data quality tasks. Proactive tasks help you identify and filter out bad data *before* you load a data set into your data lake." ### Rules that matter for a healthcare master DQDL rule types are the enforcement point for everything matching depends on. The ones that earn their place here: | Concern | Rule shape | Why it protects matching | | --- | --- | --- | | Identifier completeness | Completeness on MRN and assigning-system columns | A null system URI gives false links | | Identifier format | ColumnValues matching an NPI or MRN pattern | Malformed identifiers match nothing | | Uniqueness within source | Uniqueness on the source primary key | Duplicate Unique IDs are rejected | | Birth date plausibility | ColumnValues within a sane date range | A default date becomes a mass false match | | Referential integrity | Referential integrity between two datasets | Orphan provider references | | Volume sanity | RowCount, or a dynamic rule on a trailing average | A truncated feed shrinks the master | | Freshness | FileFreshness on the landed objects | A stalled feed looks like deletions | Two capabilities worth naming. Anomaly detection is ML-based and generally available — the release notes record "Anomaly Detection and Dynamic Rules are now generally available" in August 2024 — and dynamic rules allow thresholds relative to history, such as a row count compared with an average of the last ten runs. That is the right tool for "this feed looks wrong today" without hand-maintained thresholds. ### The limits, stated - "Data quality rules can't evaluate nested or list-type data sources." FHIR-shaped data is deeply nested, so flatten before evaluating. This is a real pipeline stage, not a footnote. - 2,000 rules per ruleset; 65 KB per ruleset. Split by entity and source. - "there is a limit of 100,000 statistics per account, and these statistics will be retained for a maximum of two years." Long-horizon quality trending needs your own store. - Billing is per DPU-hour at $0.44, with recommendation and evaluation tasks "provisioned with ... 2 DPUs with a 1 minute minimum billing duration", and anomaly detection at "1 DPU per statistic". Quality checks inside an ETL job carry no separate charge beyond the job's runtime. ### What Glue Data Quality does not do It measures and reports. It does not remediate, it does not route an exception to a human, and it does not track whether anyone acted. It emits to EventBridge and CloudWatch, which is the hook — but the queue, the assignment, the resolution and the audit of resolution are yours. ## The steward application This is the largest single gap in the pattern and the honest answer to why licensed MDM products still sell. No service cited in this set provides a steward workbench. Built properly it is months of work, and it is a PHI-handling application with its own authentication, authorisation and audit obligations. ### Why a human is unavoidable Both matchers produce assertions that need adjudication. Entity Resolution's ML matching returns match groups with "an associated confidence score of 0.0-1.0", which is precisely an invitation to a threshold and a review band. HealthLake's resource matching is explicit that review is the intended workflow: "Because matching is non-destructive, you review the resulting links and decide how to act on them." And it "does not apply priority ordering or conflict resolution between identifiers: any shared, non-placeholder identifier is enough to link two resources" — so a shared placeholder-like identifier that slipped past filtering produces links only a human will catch. In healthcare the asymmetry of error is severe. A false merge joins two patients' clinical histories and is a patient-safety event. A false split is a care-coordination failure. Neither should be decided by a threshold alone. ### The screens | Screen | Purpose | | --- | --- | | Review queue | Candidate matches in the uncertainty band, prioritised by risk | | Side-by-side compare | Two or more candidates, attribute by attribute, with source and date | | Attribute override | Pick a winning value per attribute, with a mandatory rationale | | Merge and unmerge | Join or split masters; unmerge must be a first-class operation | | Search | Find a master or source record by identifier, name or demographic | | Quality exceptions | Rows quarantined by Glue Data Quality, with assignment and resolution | | Decision history | Every decision on this master, by whom, when and why | | Match tuning | Effect of a proposed rule change against a labelled set, before deployment | Unmerge deserves emphasis because it is routinely under-built. A merge is easy; reversing one after six months of downstream propagation requires that you kept the pre-merge state and that every consumer can handle a master identifier splitting in two. Design the split event into the publish contract from the start. Amazon OpenSearch Service is the right engine behind the search and queue screens: steward search is fuzzy, multi-field and latency-sensitive, which is not what Aurora is for. Treat it as what it is — a second full copy of patient demographics, in scope for the BAA, with its own encryption, fine-grained access control and snapshot obligations, and a requirement that merges, splits and amendments propagate into the index rather than leaving a stale name searchable. Lake Formation governs what a steward sees in the curated zone, with permissions "enforced using granular controls at the column, row, and cell-levels"; the console enforces its own row scoping on the master. ### The decision store, and why it is an input The critical design property: a steward decision must be an *input* to the next pipeline run, not a log of the last one. Bulk processing "re-processes all data for any new matches and updates to existing matches", and a steward's verdict is not part of that job's state. Without replay, the next run silently reverses human judgement — and in a clinical setting, re-merging two patients a clinician separated is an incident. So the resolve stage runs in a fixed order: ``` match output (S3) steward_decision (Aurora) survivorship rules (Aurora) | | | v v v +-------------------------------------------------------------------------+ | 1. load match groups and match IDs | | 2. resolve match IDs to master_id via the crosswalk | | 3. APPLY OVERRIDES: reject_match, merge, split <- human wins | | 4. evaluate survivorship per attribute on what remains | | 5. write master_attribute versions; bump master_entity.version | | 6. emit change events to EventBridge | +-------------------------------------------------------------------------+ ``` Step 3 before step 4, always. The whole stage must be deterministic and idempotent: the same match output plus the same decision store must produce the same master, so that reprocessing is safe and a disputed value can be explained from stored inputs rather than from a log. ### Measuring match quality You cannot tune what you do not measure, and no service measures this for you. Build a labelled evaluation set — a few thousand record pairs adjudicated by humans — and run it on every rule change, reporting precision, recall and the volume landing in the review band. Be clear about what that set is: real patient pairs, so the evaluation harness is a PHI workload. It and the match tuning screen belong in the Data OU under production controls, not in a lower environment; only synthetic data goes to non-production. This is the artefact that makes a rule change a decision rather than a guess, and in healthcare it is also the evidence that the matching configuration was validated. Keep it versioned next to the rules. Two levers the documentation hands you for the tuning itself. Rule order matters, because the default waterfall means "records matched at a higher rule level are excluded from subsequent rules", and AWS advises you "order your rules to find more precise matches first". And rule-based output carries the rule number, so "the rule number can demonstrate the precision of each match group such that rule one is more precise than rule two" — which lets you set a review threshold per rule rather than one threshold for everything. ## The reference data gap One further absence. A healthcare master depends on code sets that change on a published schedule: ICD-10-CM, SNOMED CT, RxNorm, NPI registry extracts, state licence files, postal reference data. Comprehend Medical links to some of these — and pins a version, naming "the 2021 version" of ICD-10-CM on its page — but nothing cited here manages the lifecycle of your own reference data: versioning, effective dating, approval and the re-evaluation of golden records when a code is retired. That is a modest but real build, and it is the kind of thing that is forgotten until the first annual code update breaks a survivorship rule. --- You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 131 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/aws/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.