Data and analytics parity — BigQuery, Dataflow and Dataproc against Redshift/EMR/Glue and Fabric/Databricks/Synapse
Counterpart names, then the three differences that change an estimate: BigQuery's separated compute and storage against a cluster-shaped model, the inputs a warehouse estimate actually needs, and the two places a lift genuinely exists — Beam pipelines and Hadoop jobs.
markdownguidev1C8VN-YJS7stage: designtrust: officialfamily: dataservice: bigqueryinterface: anydata paritybigqueryredshiftsynapsefabricdatabricksdataflowapache beamEMRdataprocmanaged service for apache sparkgluestream analyticsslotsbytes scanned
Data and analytics parity
1. The names
| Google Cloud | AWS | Azure |
|---|---|---|
| BigQuery | Amazon Redshift | Fabric Data Warehouse, Azure Databricks; Synapse Analytics (per Google's table) |
| BigQuery for log/time-series work | — | Azure Data Explorer, Fabric Real-Time Intelligence (Microsoft's own second row) |
| Dataflow | Kinesis Data Streams with Managed Flink; Glue streaming | Azure Databricks, HDInsight; Stream Analytics |
| Dataproc / Managed Service for Apache Spark | Amazon EMR | Azure Databricks, Fabric Data Engineering |
| Cloud Data Fusion | Glue (visual ETL) | Data Factory, Data Factory in Fabric |
| Dataplex Universal Catalog | Glue Data Catalog, DataZone | Microsoft Purview |
| Looker, Looker Studio | QuickSight | Power BI |
Microsoft's table maps BigQuery twice — once as a warehouse, once as a log and time-series engine. That is a fair description of how BigQuery is used, and it means a "BigQuery replacement" estimate on Azure may need two products depending on which half of the usage you are moving.
2. The warehouse model is different, so a node count does not convert
| Google Cloud | AWS Redshift | Azure | |
|---|---|---|---|
| compute and storage | independent — "operate independently of each other" | coupled in provisioned clusters; separated in Serverless and with RA3-managed storage | varies by product |
| the compute unit | slot | node, or RPU in Serverless | capacity unit / vCore, by product |
| how compute is bought | on-demand per query, or capacity in Standard / Enterprise / Enterprise Plus editions | provisioned nodes or serverless RPUs | provisioned or serverless, by product |
| storage billing | four models: active, long-term, physical, logical | managed storage per TB | per TB by product |
| the refusal that reveals the model | billingTierLimitExceeded — an on-demand query exceeding tier 100 | — | — |
So the estimate's inputs change. A cluster-shaped estimate is "how many nodes, running how long". A BigQuery estimate is two independent numbers:
- bytes stored, and under which of the four storage models — the physical-against-logical choice turns on
how compressible the data is, and is routinely left at a default in a migrated estate;
- compute, as either bytes scanned per query per day (on-demand) or concurrent slot demand
(capacity).
Neither is derivable from a node count. Both are measurable before you migrate — see
guides/data/query-bigquery-from-typescript.md §"For a migration" for the smallest honest benchmark. An
estimate without them is a guess, and the direction of the error is unknowable.
3. Where a lift genuinely exists
This family has the two cleanest lifts in the whole hub, and they are worth finding because almost nothing else in a cloud migration is this cheap:
| Source | Target | Why it is a lift |
|---|---|---|
| Apache Beam pipelines anywhere | Dataflow | Dataflow is a Beam runner. Java, Python and Go SDKs. A Beam pipeline moves between runners by changing the runner |
| Spark, Hadoop, Hive, Pig jobs (EMR, HDInsight, on-premises) | Dataproc | the same open-source frameworks, "frequently updated" |
| HBase | Bigtable | the "Apache HBase library for Java" (reference/databases/bigtable.md §5) |
And the converse, which is just as useful in an estimate: a pipeline written against Kinesis Managed Flink or Stream Analytics is not a Beam pipeline and does not lift onto Dataflow. Checking which framework the source pipelines are written in is a ten-minute task that can move a migration estimate by months.
4. Two Google Cloud facts that change an operating model, not just a cost
- Dataproc clusters create, scale and shut down in "90 seconds or less, on average". So clusters can be
ephemeral — per job, not per environment. A Hadoop estate sized for peak and running continuously becomes cluster-hours instead of cluster-count. This is frequently the largest saving in a Hadoop migration, and it requires changing how jobs are launched, not just where they run.
- Dataflow is exactly-once by default; Pub/Sub is at-least-once by default. The pair is coherent — the
transport may duplicate and the pipeline deduplicates — but it is the opposite default pairing from several source platforms, so review what your pipeline is relying on rather than assuming it carries over.
5. Pricing dimension (never a price)
| Google Cloud | AWS | Azure | |
|---|---|---|---|
| warehouse compute | bytes scanned (on-demand) or reserved slots (capacity) | node-hours, or RPU-hours | capacity units / vCore-hours, by product |
| warehouse storage | per GB-month under one of four models | per TB-month managed storage | per TB-month |
| streaming/batch pipelines | worker vCPU, memory and the Shuffle/Streaming Engine dimensions | worker or KPU hours | DBU or streaming-unit hours |
| Spark/Hadoop | cluster-hours (or serverless units) plus the underlying VMs | EMR per-instance plus EC2 | DBU-hours plus VMs |
The structural point: BigQuery on-demand is the only one of these metered on data scanned rather than on time. That single difference is why a warehouse comparison needs a query profile. Two estates with identical data volumes and different query patterns can land on opposite sides of the comparison.
6. What this guide cannot tell you
Whether your SQL ports. Dialect differences between Redshift, T-SQL, Spark SQL and GoogleSQL are real and
per-query, and no mapping table addresses them. That is assessment work
(reference/migration/database-migration-service.md, guides/migration/assess-before-you-move.md).