Release note — data and analytics family, 2026-10-21
What the data family covers at first release, the product rename it discovered, the two clean lifts it documents, and what is known to be missing — including the Dataflow and Dataproc error surfaces.
markdownreferencev169XG-HZ89stage: designtrust: officialfamily: dataservice: bigqueryinterface: anychangelogdataanalyticsrelease notebigquerydataflowdataprocmanaged service for apache spark
Data and analytics family — first release
Added
| Document | What it carries |
|---|---|
reference/data/bigquery.md | separated compute and storage, slots, two compute models, four storage models, four load paths |
reference/data/dataflow.md | Apache Beam, one model for batch and streaming, the named features, exactly-once by default |
reference/data/dataproc.md | the rename, the frameworks, three deployment shapes, the 90-second figure |
reference/data/errors.md | the BigQuery reason strings, the retryable set, and the three rows that are the cost model talking |
guides/data/query-bigquery-from-typescript.md | a keyless client, parameterised queries, and exactly the documented retries |
guides/parity/data.md | why a node count does not convert, and where a lift genuinely exists |
What this release discovered
The page served at the Dataproc overview URL is titled "Managed Service for Apache Spark." Both names are
live — URLs, IaC resource types and CLI surfaces still say dataproc — and a document is recorded saying so,
because an agent that assumes one name is stale will mistrust correct documentation.
The two clean lifts
- Apache Beam pipelines onto Dataflow — Dataflow is a Beam runner, so a Beam pipeline changes runner.
- Spark, Hadoop, Hive and Pig onto Dataproc — the same frameworks. With HBase going to Bigtable and HDFS
to Cloud Storage, a Hadoop estate decomposes rather than being rewritten.
Both are recorded with their converse, which matters as much: a Flink or Stream Analytics pipeline does not lift onto Dataflow.
Absences recorded rather than filled in
- No vendor error table was read for Dataflow or Dataproc, so none is invented. The error catalogue says so
and gives the two facts from their own pages that carry the most diagnostic weight.
- The Dataproc overview does not state whether clusters are zonal or regional, and names no image versions.
- The BigQuery introduction names the storage billing models without stating thresholds; no figure is
quoted.
Known missing
- BigQuery partitioning, clustering, materialized views and reservations/assignments.
- BigQuery ML, BigQuery Omni, and external tables.
- Dataflow pipeline options, Dataflow Prime detail, and the Google-provided template catalogue.
- Dataproc Serverless configuration and image-version policy.
- Cloud Data Fusion, Dataplex Universal Catalog, Looker.