Storage Transfer Service — S3, Azure Blob, HDFS and POSIX sources, scheduled or event-driven, and the 1 TiB threshold
The data-movement service for object and file storage. The sources — including Amazon S3, Azure Blob Storage and on-premises POSIX and HDFS — the scheduled and event-driven modes, the agents file systems need, and the vendor's own statement about the size of transfer it is optimised for.
markdownreferencev1WZ6Z-Y1K6stage: deploytrust: officialfamily: migrationservice: storage-transfer-serviceinterface: anystorage transfer serviceAmazon S3S3-compatibleAzure Blob StorageData Lake Storage Gen2HTTP listHDFSPOSIX file systemtransfer agentevent-driven transferscheduled transfermetadata preservation1TiB
Storage Transfer Service
1. What it is, and the size it is for
"Storage Transfer Service enables seamless data movement across object and file storage systems."
Optimised for transfers exceeding 1 TiB, handling automation, scaling and network optimisation.
The 1 TiB figure is a useful threshold in both directions. Below it, a gcloud storage copy is usually
simpler. Above it, the things this service does — retrying, parallelising, validating, resuming — are the
things you would otherwise write badly.
2. The sources — and two of them are the other clouds
| Source | Note |
|---|---|
| Amazon S3 and S3-compatible storage | the AWS path, and anything speaking the S3 API |
| Microsoft Azure Blob Storage | the Azure path |
| HTTP/HTTPS URLs | publicly accessible, from a list |
| Cloud Storage buckets | bucket-to-bucket, including across locations |
| on-premises POSIX file systems and HDFS | agent-based |
HDFS as a first-class source matters for a Hadoop migration: the data moves with this service while the jobs
move to Dataproc (reference/data/dataproc.md §5).
3. Scheduled, or event-driven
Both are supported. Event-driven transfers work from Cloud Storage, AWS S3, and Azure Blob Storage / Data Lake Storage Gen2.
That is the facility that turns a migration into a continuous mirror: the source keeps receiving writes and they keep arriving at the destination. Which is how a large object estate cuts over with a short window — the bulk moved on a schedule, the tail arriving by event, and the cut-over a DNS or configuration change rather than a copy.
4. Agents, for file systems
Transfers to and from on-premises file systems need self-hosted transfer agents, which exist to give you network routing control and bandwidth management.
Bandwidth management is the operationally important half. An unthrottled transfer out of a production data centre is an outage with a good explanation. Size and throttle the agent fleet as part of the plan.
5. What is and is not stated
The overview mentions metadata preservation and data integrity validation. It does not detail delete-after-transfer or overwrite behaviour, so neither is described here.
Both of those are destructive-adjacent settings and must be read from the transfer-job configuration reference before a job runs. A mis-set overwrite or delete option on a multi-terabyte transfer is not a recoverable mistake.
6. For an estimate
| Line | Note |
|---|---|
| the transfer itself | this service |
| source-side egress | charged by the source cloud. On a large estate this is frequently the single largest cost in the project, and it is not on any Google Cloud price list |
| destination storage class | choose it knowing the minimum storage durations — ARCHIVE commits 365 days (reference/storage/cloud-storage.md §1). Landing a migration directly in a cold class starts that clock |
| validation | budget time to verify, not just to copy |
The egress row is the one to raise first with anyone asking for a number. See
guides/parity-and-migration-method.md, which treats egress as a first-class line.