Cloud Run suddenly not respecting minimum containers configuration

[flagbug]

Hey there, today at around ~02:00 AM CEST Cloud Run suddenly scaled our service down to a single container, even though the minimum instances in our service configuration is set to 2. This caused requests to be dropped, since a single container wasn't able to serve our load properly

For some reason the Cloud Run autoscaler then didn't start any new instances anymore, until we mitigated the problem by creating a new revision with the exact same original settings at ~08:37 CEST. After that, new instances were created and the minimum of 2 instances is running again.

Has anyone else experienced this problem?

[Marramirez]

Hi @flagbug,

Welcome to the Google Cloud Community!

In Cloud Run, there is a behavior called Scale to zero. If there are no incoming requests to your service, even the last remaining instance will be removed. To make sure this doesn't happen, you have to configure Cloud Run to keep a minimum amount of instances active.

In your experience, you already configured Minimum Instances, but it scaled down below the minimum. This might mean that you're having Idle Instances. 

For example, if min-instances is 10, and the number of active instances is 0, this just means that the number of idle instances is 10. When the number of active instances increases to 6, then you'll have 4 idle instances.

To learn more about Idle instances and minimizing cold starts, you can read more in this documentation.

Let me know if it helped, thanks!

[mslarkin]

This is absolutely not expected behavior.  There are a few scenarios that might cause you to briefly drop below the configured min instances (and we are actively working on reducing/eliminating that behavior), but a new instance should have started immediately.  The best way to understand exactly what happened is to open a Support case, so that we can review the logs and relevant config data.

[rachelt1]

+1 on what mslarkin mentioned. Please open a Support case so we can investigate.

[kevimunis]

We've seen this same behaviour with min_instances set to 1. Have discussed it at length with google support, but we've still yet to have this acknowledged as a bug (case 48056968 in case anyone wants to investigate). We have quite detailed logging with a number of instances of cold starts when requests come in but no instance is available, and a cold start is triggered.

As a workaround, we've had to resort to using a Datadog synthetic test which polls our service every minute. That has resolved the symptoms for us (we no longer get cold starts).