Release note — data and analytics family, 2026-10-21

What the data family covers at first release, the product rename it discovered, the two clean lifts it documents, and what is known to be missing — including the Dataflow and Dataproc error surfaces.

markdownreferencev169XG-HZ89stage: designtrust: officialfamily: dataservice: bigqueryinterface: anychangelogdataanalyticsrelease notebigquerydataflowdataprocmanaged service for apache spark

Created 21 Oct 2026 · Modified 23 Oct 2026 · 1 version

Data and analytics family — first release

Added

DocumentWhat it carries
reference/data/bigquery.mdseparated compute and storage, slots, two compute models, four storage models, four load paths
reference/data/dataflow.mdApache Beam, one model for batch and streaming, the named features, exactly-once by default
reference/data/dataproc.mdthe rename, the frameworks, three deployment shapes, the 90-second figure
reference/data/errors.mdthe BigQuery reason strings, the retryable set, and the three rows that are the cost model talking
guides/data/query-bigquery-from-typescript.mda keyless client, parameterised queries, and exactly the documented retries
guides/parity/data.mdwhy a node count does not convert, and where a lift genuinely exists

What this release discovered

The page served at the Dataproc overview URL is titled "Managed Service for Apache Spark." Both names are live — URLs, IaC resource types and CLI surfaces still say dataproc — and a document is recorded saying so, because an agent that assumes one name is stale will mistrust correct documentation.

The two clean lifts

  1. Apache Beam pipelines onto Dataflow — Dataflow is a Beam runner, so a Beam pipeline changes runner.
  2. Spark, Hadoop, Hive and Pig onto Dataproc — the same frameworks. With HBase going to Bigtable and HDFS

to Cloud Storage, a Hadoop estate decomposes rather than being rewritten.

Both are recorded with their converse, which matters as much: a Flink or Stream Analytics pipeline does not lift onto Dataflow.

Absences recorded rather than filled in

  • No vendor error table was read for Dataflow or Dataproc, so none is invented. The error catalogue says so

and gives the two facts from their own pages that carry the most diagnostic weight.

  • The Dataproc overview does not state whether clusters are zonal or regional, and names no image versions.
  • The BigQuery introduction names the storage billing models without stating thresholds; no figure is

quoted.

Known missing

  • BigQuery partitioning, clustering, materialized views and reservations/assignments.
  • BigQuery ML, BigQuery Omni, and external tables.
  • Dataflow pipeline options, Dataflow Prime detail, and the Google-provided template catalogue.
  • Dataproc Serverless configuration and image-version policy.
  • Cloud Data Fusion, Dataplex Universal Catalog, Looker.

Read this document as markdown — the same bytes an agent receives.