This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Log in to ask a question

How to maximize download from Google Cloud Storage to VM?

Posted 2 weeks ago
vedantroy-genmo
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

I created a VM in the same region as the my Google Cloud storage bucket, but when downloading from the bucket using rclone, the download speed is only ~ 263MiB/s. How can I get faster download speeds?

I am using Tier 1 networking--but that doesn't seem to be helping.

5 3 140
Topic Labels
3 REPLIES 3

Posted on --/--/---- --:-- AM
MaxImbrox
Silver 2
Silver 2
Reply posted on --/--/---- --:-- AM

Hi @vedantroy-genmo,

sometimes the transfer speed is not related only to network parameters, but also to the performances of the disk attached to the VM. Have you verified if your disk have something like a max value of throughput? (https://cloud.google.com/compute/docs/disks/performance#performance_factors)

Besides this, have you tried to use the gsutil command through Google APIs to transfer the files?

Posted on --/--/---- --:-- AM
alexmoore
Staff
Reply posted on --/--/---- --:-- AM

There is some guidance on this page about how Google Cloud Storage scales:

https://cloud.google.com/storage/docs/request-rate

@MaxImbrox commented to use gsutil, which is a good suggestion, I would add that you may also want to explore "gcloud storage cp" as this has been optimised even further, check out this blog post for more details:

https://cloud.google.com/blog/products/storage-data-transfer/new-gcloud-storage-cli-for-your-data-tr...

All the best,

Alex

Posted on --/--/---- --:-- AM
MaxImbrox
Silver 2
Silver 2
In response to alexmoore
Reply posted on --/--/---- --:-- AM

@alexmoore very good tip for next times! 🙂