Cloud Run container stops processing when running parallel Fastapi background tasks

[Djay9213]

so I have a local running Fastapi app that upon receiving a request, starts a background task (using Fastapi background tasks) and upon completion writes the response to the database. This works well when the app is running locally when I start multiple background tasks around the same time with my requests and it successfully writes to my cloud/supabase database. However, when I deploy this same app to my cloud run (Note I use 'cpu always allocated' instance with a 6cpu and 6 gb ram) 1 background task seem to be running okay, but when I start multiple tasks simultaneously with requests, it seems to stop processing after a while and nothing seems to be happening, it stops both background tasks, ultimately the cloud run container shuts down and database is also not populated. I looked at my container usage and my cpu and ram are sufficient enough for this, and even if not I expected it to start multiple instances to process. Also note I don't see any logs in my Cloud run console and have set adequate concurrency limit (40). I thought it might have to do with my supabase database connection, but considering it works locally with the same database it seems like I'm missing something on how cloud run seems to be working with fastapi background tasks.

Note: It does not at any time exceeds cpu or memory, my timeouts are 3600 seconds which is quite enough. I tried adjusting the concurrency settings as well (set it to 1) but still this issue. I keep sending a polling request from the client for the status on the background task and that request I do see in my logs for quite sometime.

[ronnelg · Accepted solution]

Hi @Djay9213, 

Welcome to Google Cloud Community!

It seems that your container image is not designed for handling multiple requests at the same time, you can try to use the default value of Maximum concurrent requests per instance which is 80, if it still didn’t work, try to use a different concurrency value starting with a lower concurrency, and then moving it up. Starting with a concurrency that is too high could lead to unintended behavior due to resource constraints. You can also try to increase the maximum number of container instances for your service and see if it helps.

To optimize your maximum concurrent requests per instance setting, see development tips for tuning concurrency.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[Djay9213]

Hi @ronnelg many thanks for your response. Yes, that seemed to be the case, I adjusted the concurrency settings and increasing cpu for my container and that seems to resolve the issue.