cloudrun: 500 error - The request failed because the instance could not start successfully.

[ramkrivas]

We notice lot of 500 error - The request failed because the instance could not start successfully errors in the logexplorer of our cloudrun services.  no additional informations and also we could not see any relevant logs from application side. 

We guess that during the spike time this particular error gets logged more from cloudrun.

Here's the cloudrun service setup:

• concurrency configured per container instance is 8
• Auto scaling: min: 2, max : 10
• Liveness probe : Initial delays: 5s, timeout: 1s, failure threshold: 3
• Healthness probe: Initial delays: 10s, timeout: 5s, failure threshold 3

Can anyone has an idea what could be the issue and why cloudrun logs this error?

[Marramirez]

Hi @ramkrivas,

Welcome to the Google Cloud Community!

You can look into this Stack Overflow Post as you might have the same problem, as well as the explanation to why you are getting that error.

If you are using Cloud SQL, then it could be the cause of Cloud Run auto-scaling maximum instance limit and Cloud SQL's connection limit. You can see your Quotas and Limits here.

Under Connection Limits, there is a scenario where Cloud Run Services are limited to 100 per Cloud SQL Database. This limit applies per service instance. This means that there are 100 Cloud SQL connection per Cloud Run Instance, as it scales the total number grows.

You can limit the maximum number of connections used by an instance by using a connection pool. 

If the above option doesn't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!

[ramkrivas]

@Marramirez - we don't use Cloud SQL, so definitely no need to think about that perceptive. Here are our observations so far :

• This errors getting logged during peak traffic and while cloudrun tries to scale up. 
• Also, If we disable the liveness probe check , then this error is not arise back. so I believed liveness check is the trouble maker here. However, I don't see any logs related to memory issues.