---
cuppy_hub: gcp
cuppy_path: reference/data/dataflow.md
cuppy_section: docs
cuppy_doc: FQNB-BTB9
cuppy_doc_uuid: 80075683-9cc8-4a7d-acfc-8df54e44672a
cuppy_permalink: https://cuppy.io/gcp/d/FQNB-BTB9
cuppy_version: 1
cuppy_updated: 2026-10-23T16:16:52.575Z
cuppy_canonical: https://cuppy.io/gcp/docs/reference/data/dataflow
cuppy_mcp: https://cuppy.io/mcp
cuppy_body_sha256: 0b37e0828ef5a57368e67be0950d2ca4d5d3a8dda0132c58ada7b940e2ddd977
# cuppy_body_sha256 is over the document body ALONE, excluding both generated blocks.
---


# Dataflow

## 1. What it is

> "A Google Cloud service that provides unified stream and batch data processing at scale."

It runs **Apache Beam** pipelines, with **Java, Python and Go** SDKs, and Google provisions and manages the
worker VMs.

**"Unified" is the load-bearing word.** Batch and streaming use the **same programming model**, so one pipeline
definition can be run either way. For a migration that is the difference between porting one codebase and
porting two.

## 2. The features, and what each is for

| Feature | What it does |
|---|---|
| **Streaming Engine** | low-latency streaming |
| **Dataflow Prime** | **vertical** autoscaling |
| **Shuffle** | the shuffle step for batch jobs, moved off the workers |
| **horizontal autoscaling** with **dynamic work rebalancing** | worker count follows the work, and work moves between workers mid-job |
| **templates** | Google-provided and custom, so a pipeline can be launched without building the code |

**Dynamic work rebalancing is the one with no obvious counterpart elsewhere**: a straggler's work is taken and
redistributed while the job runs, instead of the job waiting for it. That is why a Dataflow batch job's tail
behaves differently from a fixed-partition framework's.

**Google-provided templates** are worth checking before writing anything: common pipelines —
Pub/Sub to BigQuery, Cloud Storage to Bigtable, and so on — already exist, and a template is a launch rather
than a build.

## 3. The processing guarantee

**Exactly-once processing by default**, with an **at-least-once mode available**.

Note the asymmetry with Pub/Sub, which is **at-least-once by default** with exactly-once configurable
(`reference/messaging/pubsub.md` §3). The two defaults point in opposite directions, which matters when
Pub/Sub feeds Dataflow: the transport may deliver twice and the pipeline is responsible for the
deduplication — which, in the default mode, it does.

Choosing at-least-once mode in Dataflow is a **throughput and latency** decision taken knowingly, and it
hands the idempotency requirement back to your logic.

## 4. Location

Dataflow has **regional** endpoints. So a pipeline is a regional thing, and its workers, its sources and its
sinks all have locations that should agree — a cross-region read is both slower and separately billed.

## 5. Before you write a pipeline

Two checks that routinely remove the pipeline entirely:

1. **a Pub/Sub subscription type** — BigQuery, Bigtable or Cloud Storage — may already do what the pipeline
   was going to do, with no code (`reference/messaging/pubsub.md` §2);
2. **a Google-provided template** may already be the pipeline.

A migrated architecture frequently carries a streaming job whose entire function is one of those two. Finding
them is part of the estimate, not an optimisation afterwards
(`guides/parity/data.md`).


---
<!-- cuppy -->
You are reading one file. The same content is free over MCP at https://cuppy.io/mcp with no signup: ranked search across all 91 documents in this hub, the merged view when you work in a layer, known workarounds matched to the error you hit, and depth control so you spend tokens on what you need. Start with the `_hello` tool. How to connect: https://cuppy.io/gcp/ Something wrong in this document? Report it with the `session feedback` tool on the same endpoint.
<!-- /cuppy -->