On-prem to GCS transfer via gcloud CLI - Can gcloud storage command run for long hours?

[dheerajpanyam]

I intend to use a simple gcloud storage CLI to transfer 500GB data from on-prem to GCS. Is it tested to run for long hours , maybe 8-10 hours.  Hope it doesn't get stalled, crash etc?

[greb · Accepted solution]

Hi @dheerajpanyam,

Transferring 500GB of data to GCS via the gcloud storage CLI may take 8-10 hours, influenced by network speed and system performance. Although the CLI is strong, extended transfers might encounter problems such as network timeouts or system disruptions.

You may follow some workaround to ensure reliability:

1. Use rsync instead of cp for large-scale transfers (gcloud storage rsync -r SRC DST).
2. Enable resumable transfers to restart incomplete uploads if interrupted.
3. Increase parallelism with --num-threads=4 (adjust based on system capacity).
4. Use a GCE VM in the same region as GCS to improve transfer speed if needed.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[dheerajpanyam]

Hi @greb . My use case  on premises  data transfer  to GCP . So the GCE VM, colocation does not apply here but the remaining 3 options are very good and should speed up data transfer. Thank you so much.

[sinjini]

Hey, @greb ! Looks, like gcloud handles parallelism internally, I could  not find any flag for threads in the docs. Also, gcloud storage took 30mins to transfer 15gb data into the bucket, which is closer to my region, with 7.5MiB/s throughput. Let me know if there is anything I am missing out.  Here is the command I ran for testing 

gcloud storage rsync "$SOURCE_DIR" "$GCS_BUCKET" --recursive --verbosity=info