Cloud Run Background Tasks: Preventing Instance Termination During Processing

[Asher_Yo]

I'm implementing a background processing system using Cloud Run with the following flow:

• HTTP request is received
• Request triggers a background process
• Process updates our database

To ensure continuous availability, I've set min-instances=1.

My concern is about instance termination during scale-down events. If Cloud Run scales up to handle traffic and then scales down, how can I ensure that instances running active background processes aren't terminated before completion?

Would the active CPU utilization from the background process prevent instance termination, or do I need additional safeguards?

Any best practices or configuration recommendations would be appreciated.

@knet I'd appreciate your expertise on this 🙂

[rmrf]

No, this is not possible in Cloud Run services. This is why Cloud Run Jobs was created which allows "run to completion".

https://cloud.google.com/run/docs/create-jobs

[Asher_Yo]

Hi thank you for the response.

can please you elaborate on what is not possible?

see below from the cloud run documentation -

"However, if needed, you can change this default to specify an instance to be kept idle or "warm" using the minimum instances setting. If you are using CPU outside of requests, you should set minimum instances equal to 1."

thanks again

[knet]

Hi Asher_Yo,

How long is the background process?

For best reliability, I'd recommend kicking off the background process as a Cloud Run jobs. This way, you can keep the service itself on the "request-based billing" model so it will be able to scale down between requests, and the background jobs will run to completion independently of the service. This will also work well as your service scales.

Now, if your background process is just a few seconds long, this isn't very economical because jobs are billed for a 1 minute minimum. 

Another option is to do what you're doing: Set the service to instance-based billing, which keeps the CPU unthrottled. Set min instances to 1. And, crucially, catch the SIGTERM signal, which gives you 10 seconds to wrap up your background processing before the instance is scaled down. 

Even with min instances=1 and instance-based billing, you will encounter instance restarts (a few a week, on average) and, if you ever scale above 1 instance, instance shutdown events due to scale down. This is what rmrf was referring to. So, you need to be prepared for that. There is no way to run Cloud Run in some sort of "never restart please" mode.