cloud bucket blob download is very slow in cloud r...

ghongane21 · 07-23-2023 06:41 AM

I am using python sdk to download large (1Gb) object from cloud storage bucket inside cloud run object. cloud run has 8 GB memory and 4 cpus. I tried it with various chunk size and various worker count.
Below is my code:

```

  bucket = storage_client.bucket('mybucket')
  chunk_list = [33554432, 52428800, 78905344, 104857600]
  work_list = [4, 8, 16, 22, 32, 48]
  for chunk in chunk_list:
    for worker in work_list:
      blob = bucket.blob('my1GbBlob')
      print('download started: ', 'worker:', worker, 'chunk_size: ', chunk)
      start_time = datetime.now(timezone.utc)
      transfer_manager.download_chunks_concurrently(blob, '/tmp/myTmpFile_' + str(worker) + '_' + str(chunk), chunk_size=chunk, max_workers=worker)
      deltaTime = (datetime.now(timezone.utc) - start_time)
      executionTimeMilliSec = (round(deltaTime.total_seconds()*1000))
      print('download completed: ', worker, chunk, executionTimeMilliSec)
      os.remove('/tmp/myTmpFile_' + str(worker) + '_' + str(chunk))

```

I tried this code with with 2 cpu and 8 GB memory in cloud run where it was using 100% CPU so i switched to cloud run with 4 cpu and 8 Gb where cpu utilization is normal 55 % Speed is never goes above 13 sec to download 1 GB file.

How can i achieve more download speed ?

Note: 1. this is cloud function gen2 environment.

Note: 2. the same script gives speed of 4 seconds in default cloud shell machine
Note: 3. Machine is in us-central1 and bucket is in us multi-region. My cloud shell is located in asia-southeast1

Marramirez

Hi @ghongane21,

Welcome to the Google Cloud Community!

You can try the following options to have faster download speed:

You can Use Minimum Instances to reduce cold start. For example, using a minimum instance of 1 means that you don't have to start a new instance because your service is ready to receive a number of concurrent requests configured.
Set your Cloud Run service CPU to be always allocated. This allows background activities to run outside of requests while having CPU access.
Check the best practices for Optimizing Performance.
If the above options don't work, you can contact Google Cloud Support to further look into your case.

Let me know if it helped, thanks!

ghongane21

@Marramirez ,

Thank you for your reply.
I already tried option 1 and 2 and most of the things from. with just 1 concurrency and 8 GB 4CPU configuration. My task is not background task, it is running in same request context.

Any suggestion on base image for container and will it affect multithreading. I used (python:3.10-buster) ?
OR it's something related to 32 MB request response limit of cloud container?
Is there document around baseline speed of downloading cloud bucket object in cloud run ?

This week we are exploring speed on S3 + Lambda. Let's see. If it wont work will try your option 4.

ghongane21

UPDATE:
@Marramirez
We deployed same container on cloud run , local machine, GKE.
Local machine and GKE achieved our required speed 245 MB/s.
Cloud run stuck at 67 MB/s any reason why ?

cloud bucket blob download is very slow in cloud run