---
cuppy_hub: gcp
cuppy_path: reference/data/dataproc.md
cuppy_section: docs
cuppy_doc: 7GTP-8NH9
cuppy_doc_uuid: faa0d28a-0222-47b9-85c3-dea213fbc022
cuppy_permalink: https://cuppy.io/gcp/d/7GTP-8NH9
cuppy_version: 1
cuppy_updated: 2026-10-23T16:16:52.575Z
cuppy_canonical: https://cuppy.io/gcp/docs/reference/data/dataproc
cuppy_mcp: https://cuppy.io/mcp
cuppy_body_sha256: 1a7f8c1da852d0233b9de78fd2143df850400af2d75fe025ecbdb4956e43a355
# cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks.
---


# Dataproc

## 1. The name, which has moved

**The page served at the Dataproc overview URL is titled "Managed Service for Apache Spark."** The product's
own documentation now leads with that name, and the overview's framing sentence is:

> "Managed Service for Apache Spark on clusters lets you take advantage of open source data tools for batch
> processing, querying, streaming, and machine learning."

An agent should expect **both** names in the wild — URLs, IaC resource types and CLI surfaces still carry
`dataproc` — and should not assume a document, console page or API using one name is stale. Where a resource
type is `dataproc`, use `dataproc`.

## 2. What it runs

**Spark, Hadoop, Hive and Pig**, with "Spark, Hadoop, Pig, and Hive frequently updated".

This is the lift-and-shift target for an on-premises or cloud-hosted Hadoop estate: the frameworks are the same
ones, so jobs move before they are rewritten.

## 3. Three deployment shapes

| Shape | What it is |
|---|---|
| **clusters** | the traditional form: you create a cluster and submit work to it |
| **Serverless for Apache Spark** | no cluster to create |
| **Dataproc on GKE** | the workload runs on a Kubernetes cluster you already have |

The choice is about **who holds the cluster**, and it is the main cost and operations decision in this service.

## 4. The 90-second figure, and why it changes the design

> "Managed Service for Apache Spark clusters are quick to start, scale, and shutdown, with each of these
> operations taking **90 seconds or less, on average**."

That is a stated vendor figure and it has a direct design consequence: **the cluster can be ephemeral.** Create
it for the job, run the job, delete it. A long-lived cluster sitting idle between jobs is the
on-premises habit, and on a platform where creation takes a minute and a half it is simply paid-for idleness.

For a migration estimate, this is often the largest single saving in a Hadoop move: the source estate is sized
for peak and runs continuously; the target can be sized per job and exist only while the job does. **Count
cluster-hours, not cluster count.**

## 5. Where the rest of a Hadoop estate goes

A Hadoop estate rarely lands entirely on one service:

| Source component | Likely Google Cloud home |
|---|---|
| Spark and Hive jobs | **Dataproc** (clusters, serverless, or on GKE) |
| HBase | **Bigtable**, which integrates with the "Apache HBase library for Java" (`reference/databases/bigtable.md` §5) |
| HDFS | **Cloud Storage** (`reference/storage/cloud-storage.md`) |
| the warehouse-shaped queries | **BigQuery** (`reference/data/bigquery.md`) |

Splitting the estate that way is the assessment, and it is per component.

## 6. Absences recorded

The overview page **does not state whether clusters are zonal or regional**, and references "cluster image
version lists" **without naming versions**. Neither is asserted here. Establish both before planning a
migration whose availability or framework-version requirements depend on them.


---
<!-- cuppy -->
You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 91 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/gcp/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.
<!-- /cuppy -->