gsutils vs 'gcloud storage' performance

[vitaly_il]

I learned about  'gcloud storage'  recently. According to https://cloud.google.com/blog/products/storage-data-transfer/new-gcloud-storage-cli-for-your-data-tr... it should be faster than gsutils, at least for copy.
But I noticed that 'ls' operation faster in gsutils, 'gcloud storage' takes about 30% more time. In both cases I need to generate a list of .zip files under 'Unprocessed' folder. Did I miss something, or this is a situation for now?

1) gsutil ls -r gs://my-bucket/** | grep Unprocessed|grep '\.zip$'

2) gcloud storage ls gs://my-bucket/**/Unprocessed/*.zip

[anjelisa · Accepted solution]

Hi @vitaly_il,

Gcloud storage as you now probably know is a new set of Cloud Storage commands in Cloud SDK that has been innovated in order to be fast by default especially when it comes to data transfer rates and is created as an improvement to Gsutil. These tests prove the difference in performance between the two by testing the download and upload speed.

When transferring 100 files that were 100MB in size, gcloud storage is 79% faster than gsutil on download and 33% faster on upload using a parallel composite upload strategy. See Figure 1. With a 10GB file, gcloud storage is 94% faster than gsutil on download and 57% faster on upload. See Figure 2. These tests have been performed on Google Cloud Platform using n2d-standard-16 (8 vCPUs, 32 GB memory) and 1x375GB NVME in RAID0 in us-east4.

---

The article did not mention the difference in Listing Cloud Storage buckets and objects. To test it out, I recommend to use the same wildcard/flags, and if upon comparing that the other one is better, it's always beneficial to use what works best for you.

See helpful reference:

https://cloud.google.com/storage/docs/gsutil/commands/ls

https://cloud.google.com/sdk/gcloud/reference/storage/ls

[Chaya]

Hi,
Do you have any information on what will be different in terms of performance using gsutil or gcloud storage - when we are talking about uploading a maximum of 16 files, the maximum weight of each file is 20MB?