--- cuppy_hub: gcp cuppy_path: guides/parity/data.md cuppy_section: docs cuppy_doc: C8VN-YJS7 cuppy_doc_uuid: 93f960e6-01b6-46dc-8e15-8b7df882a4fd cuppy_permalink: https://cuppy.io/gcp/d/C8VN-YJS7 cuppy_version: 1 cuppy_updated: 2026-10-23T16:16:52.575Z cuppy_canonical: https://cuppy.io/gcp/docs/guides/parity/data cuppy_mcp: https://cuppy.io/mcp cuppy_body_sha256: a9c1a740b805c38829e8ed83366c4e60523f7fea3dbccf063e1e9878544af184 # cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks. --- # Data and analytics parity ## 1. The names | Google Cloud | AWS | Azure | |---|---|---| | **BigQuery** | Amazon Redshift | **Fabric Data Warehouse**, Azure Databricks; Synapse Analytics (per Google's table) | | BigQuery for log/time-series work | — | **Azure Data Explorer**, Fabric Real-Time Intelligence (Microsoft's own second row) | | **Dataflow** | Kinesis Data Streams with Managed Flink; Glue streaming | Azure Databricks, HDInsight; **Stream Analytics** | | **Dataproc** / Managed Service for Apache Spark | Amazon EMR | Azure Databricks, Fabric Data Engineering | | Cloud Data Fusion | Glue (visual ETL) | Data Factory, Data Factory in Fabric | | Dataplex Universal Catalog | Glue Data Catalog, DataZone | **Microsoft Purview** | | Looker, Looker Studio | QuickSight | Power BI | **Microsoft's table maps BigQuery twice** — once as a warehouse, once as a log and time-series engine. That is a fair description of how BigQuery is used, and it means a "BigQuery replacement" estimate on Azure may need *two* products depending on which half of the usage you are moving. ## 2. The warehouse model is different, so a node count does not convert | | Google Cloud | AWS Redshift | Azure | |---|---|---|---| | compute and storage | **independent** — "operate independently of each other" | coupled in provisioned clusters; separated in Serverless and with RA3-managed storage | varies by product | | the compute unit | **slot** | node, or RPU in Serverless | capacity unit / vCore, by product | | how compute is bought | **on-demand per query**, or **capacity** in Standard / Enterprise / Enterprise Plus editions | provisioned nodes or serverless RPUs | provisioned or serverless, by product | | storage billing | **four models**: active, long-term, physical, logical | managed storage per TB | per TB by product | | the refusal that reveals the model | **`billingTierLimitExceeded`** — an on-demand query exceeding tier 100 | — | — | **So the estimate's inputs change.** A cluster-shaped estimate is "how many nodes, running how long". A BigQuery estimate is **two independent numbers**: 1. **bytes stored**, and under which of the four storage models — the physical-against-logical choice turns on how compressible the data is, and is routinely left at a default in a migrated estate; 2. **compute**, as either **bytes scanned per query per day** (on-demand) or **concurrent slot demand** (capacity). Neither is derivable from a node count. Both are measurable before you migrate — see `guides/data/query-bigquery-from-typescript.md` §"For a migration" for the smallest honest benchmark. An estimate without them is a guess, and the direction of the error is unknowable. ## 3. Where a lift genuinely exists This family has the two cleanest lifts in the whole hub, and they are worth finding because almost nothing else in a cloud migration is this cheap: | Source | Target | Why it is a lift | |---|---|---| | **Apache Beam** pipelines anywhere | **Dataflow** | Dataflow *is* a Beam runner. Java, Python and Go SDKs. A Beam pipeline moves between runners by changing the runner | | **Spark, Hadoop, Hive, Pig** jobs (EMR, HDInsight, on-premises) | **Dataproc** | the same open-source frameworks, "frequently updated" | | **HBase** | **Bigtable** | the "Apache HBase library for Java" (`reference/databases/bigtable.md` §5) | And the converse, which is just as useful in an estimate: a pipeline written against **Kinesis Managed Flink** or **Stream Analytics** is *not* a Beam pipeline and does not lift onto Dataflow. Checking which framework the source pipelines are written in is a ten-minute task that can move a migration estimate by months. ## 4. Two Google Cloud facts that change an operating model, not just a cost 1. **Dataproc clusters create, scale and shut down in "90 seconds or less, on average".** So clusters can be **ephemeral** — per job, not per environment. A Hadoop estate sized for peak and running continuously becomes cluster-*hours* instead of cluster-*count*. This is frequently the largest saving in a Hadoop migration, and it requires changing how jobs are launched, not just where they run. 2. **Dataflow is exactly-once by default**; Pub/Sub is at-least-once by default. The pair is coherent — the transport may duplicate and the pipeline deduplicates — but it is the **opposite** default pairing from several source platforms, so review what your pipeline is relying on rather than assuming it carries over. ## 5. Pricing dimension (never a price) | | Google Cloud | AWS | Azure | |---|---|---|---| | warehouse compute | **bytes scanned** (on-demand) or **reserved slots** (capacity) | node-hours, or RPU-hours | capacity units / vCore-hours, by product | | warehouse storage | per GB-month under **one of four models** | per TB-month managed storage | per TB-month | | streaming/batch pipelines | worker vCPU, memory and the Shuffle/Streaming Engine dimensions | worker or KPU hours | DBU or streaming-unit hours | | Spark/Hadoop | **cluster-hours** (or serverless units) plus the underlying VMs | EMR per-instance plus EC2 | DBU-hours plus VMs | The structural point: **BigQuery on-demand is the only one of these metered on data scanned rather than on time**. That single difference is why a warehouse comparison needs a query profile. Two estates with identical data volumes and different query patterns can land on opposite sides of the comparison. ## 6. What this guide cannot tell you Whether your SQL ports. Dialect differences between Redshift, T-SQL, Spark SQL and GoogleSQL are real and per-query, and no mapping table addresses them. That is assessment work (`reference/migration/database-migration-service.md`, `guides/migration/assess-before-you-move.md`). --- You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 91 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/gcp/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.