Dataflow — Apache Beam, one model for batch and streaming, and exactly-once by default
The managed runner for Apache Beam pipelines. One programming model for batch and streaming, the named features including Streaming Engine and Prime, the autoscaling behaviour, templates, and the processing guarantee — exactly-once by default, with at-least-once available.
markdownreferencev1FQNB-BTB9stage: designtrust: officialfamily: dataservice: dataflowinterface: sdkdataflowApache BeambatchstreamingStreaming EngineDataflow PrimeShufflehorizontal autoscalingvertical autoscalingdynamic work rebalancingtemplatesexactly-once processingat-least-once mode
Dataflow
1. What it is
"A Google Cloud service that provides unified stream and batch data processing at scale."
It runs Apache Beam pipelines, with Java, Python and Go SDKs, and Google provisions and manages the worker VMs.
"Unified" is the load-bearing word. Batch and streaming use the same programming model, so one pipeline definition can be run either way. For a migration that is the difference between porting one codebase and porting two.
2. The features, and what each is for
| Feature | What it does |
|---|---|
| Streaming Engine | low-latency streaming |
| Dataflow Prime | vertical autoscaling |
| Shuffle | the shuffle step for batch jobs, moved off the workers |
| horizontal autoscaling with dynamic work rebalancing | worker count follows the work, and work moves between workers mid-job |
| templates | Google-provided and custom, so a pipeline can be launched without building the code |
Dynamic work rebalancing is the one with no obvious counterpart elsewhere: a straggler's work is taken and redistributed while the job runs, instead of the job waiting for it. That is why a Dataflow batch job's tail behaves differently from a fixed-partition framework's.
Google-provided templates are worth checking before writing anything: common pipelines — Pub/Sub to BigQuery, Cloud Storage to Bigtable, and so on — already exist, and a template is a launch rather than a build.
3. The processing guarantee
Exactly-once processing by default, with an at-least-once mode available.
Note the asymmetry with Pub/Sub, which is at-least-once by default with exactly-once configurable
(reference/messaging/pubsub.md §3). The two defaults point in opposite directions, which matters when
Pub/Sub feeds Dataflow: the transport may deliver twice and the pipeline is responsible for the
deduplication — which, in the default mode, it does.
Choosing at-least-once mode in Dataflow is a throughput and latency decision taken knowingly, and it hands the idempotency requirement back to your logic.
4. Location
Dataflow has regional endpoints. So a pipeline is a regional thing, and its workers, its sources and its sinks all have locations that should agree — a cross-region read is both slower and separately billed.
5. Before you write a pipeline
Two checks that routinely remove the pipeline entirely:
- a Pub/Sub subscription type — BigQuery, Bigtable or Cloud Storage — may already do what the pipeline
was going to do, with no code (reference/messaging/pubsub.md §2);
- a Google-provided template may already be the pipeline.
A migrated architecture frequently carries a streaming job whose entire function is one of those two. Finding
them is part of the estimate, not an optimisation afterwards
(guides/parity/data.md).