Running Google Cloud Tasks slow down Google Cloud Run service.

[noopurphalak]

Hi,

I have a Google Cloud Run service running with Django under the hood. Usually, I would use celery to carry out long background tasks but as I have learnt, this is not possible in Google Cloud Run. So, I decided to use Cloud Tasks. Let me give you an overview of the setup I have currently:

1. The docker image which runs Django through Gunicorn runs a entrypoint command

"gunicorn wsgi:application --bind 0.0.0.0:8080 --workers 4 --threads 32 --timeout 0"

2. My Cloud Run service is configured in such a way that each container has 8 GB of memory and has 4 vCPUs. So, I am spawning 1 worker per core and 8 threads per core.

3. Now, I have a cloud scheduler scheduled every hour which calls the cloud run service api which then creates several google cloud tasks (almost 3000 to be precise.)

4. I have the task queue to dispatch 1 task every second and max concurrent dispatches to 60.

With all the above setup, when I try to run the google cloud tasks, I find that the Cloud Run service slows down even though I have multiple workers and threads spawned.

If you could help me out in suggesting how I can replicate the behaviour of celery in Cloud Run service. If I cannot replicate the behaviour, how can I solve the above issue I am facing?

[knet]

Hi, 

I'm not sure what's causing this slowdown, but here are a few other ideas:

 - Instead of using Cloud Scheduler and Cloud Tasks, use Cloud Run jobs for these long-running tasks. Cloud Run jobs start a job that runs until completion. This way, you decouple the background jobs from your service - your service only serves user requests and you offload the background processing. 

 - Enable "CPU Always Allocated" on Cloud Run and set up min instances. This way, you have a few instances always running and able to carry out background tasks. Without understanding your use case too much, I'd expect this method to be less reliable (the min instances can and do restart on occasion) and more expensive (the instances run all the time; the jobs only run when there's something to run). 

My recommendation would be to try Cloud Run jobs.