Dataproc — now documented as Managed Service for Apache Spark, three deployment shapes, and clusters that start in 90 seconds
Managed open-source data processing. The renaming the documentation now carries, the frameworks it runs, the three deployment options, and the vendor's stated 90-seconds-or-less figure for create, scale and shutdown — which is the fact that changes how you use it.
markdownreferencev17GTP-8NH9stage: designtrust: officialfamily: dataservice: dataprocinterface: anydataprocManaged Service for Apache SparkSparkHadoopHivePigServerless for Apache SparkDataproc on GKEcluster image versionephemeral cluster90 seconds
Dataproc
1. The name, which has moved
The page served at the Dataproc overview URL is titled "Managed Service for Apache Spark." The product's own documentation now leads with that name, and the overview's framing sentence is:
"Managed Service for Apache Spark on clusters lets you take advantage of open source data tools for batch processing, querying, streaming, and machine learning."
An agent should expect both names in the wild — URLs, IaC resource types and CLI surfaces still carry
dataproc — and should not assume a document, console page or API using one name is stale. Where a resource
type is dataproc, use dataproc.
2. What it runs
Spark, Hadoop, Hive and Pig, with "Spark, Hadoop, Pig, and Hive frequently updated".
This is the lift-and-shift target for an on-premises or cloud-hosted Hadoop estate: the frameworks are the same ones, so jobs move before they are rewritten.
3. Three deployment shapes
| Shape | What it is |
|---|---|
| clusters | the traditional form: you create a cluster and submit work to it |
| Serverless for Apache Spark | no cluster to create |
| Dataproc on GKE | the workload runs on a Kubernetes cluster you already have |
The choice is about who holds the cluster, and it is the main cost and operations decision in this service.
4. The 90-second figure, and why it changes the design
"Managed Service for Apache Spark clusters are quick to start, scale, and shutdown, with each of these operations taking 90 seconds or less, on average."
That is a stated vendor figure and it has a direct design consequence: the cluster can be ephemeral. Create it for the job, run the job, delete it. A long-lived cluster sitting idle between jobs is the on-premises habit, and on a platform where creation takes a minute and a half it is simply paid-for idleness.
For a migration estimate, this is often the largest single saving in a Hadoop move: the source estate is sized for peak and runs continuously; the target can be sized per job and exist only while the job does. Count cluster-hours, not cluster count.
5. Where the rest of a Hadoop estate goes
A Hadoop estate rarely lands entirely on one service:
| Source component | Likely Google Cloud home |
|---|---|
| Spark and Hive jobs | Dataproc (clusters, serverless, or on GKE) |
| HBase | Bigtable, which integrates with the "Apache HBase library for Java" (reference/databases/bigtable.md §5) |
| HDFS | Cloud Storage (reference/storage/cloud-storage.md) |
| the warehouse-shaped queries | BigQuery (reference/data/bigquery.md) |
Splitting the estate that way is the assessment, and it is per component.
6. Absences recorded
The overview page does not state whether clusters are zonal or regional, and references "cluster image version lists" without naming versions. Neither is asserted here. Establish both before planning a migration whose availability or framework-version requirements depend on them.