Cloud run to cloud sql via cloud sql proxy random dropped connections

[Stijno24]

We see randomly dropped connection in quite big quantities for a few seconds. We use node a connection string like this /cloudsql/instanceId

The drops occur on random cloud sql mysql instances on random cloud run instances but frequently at the same time which does suggest some external cause.

Reading logs in cloud run 

2022-11-04 21:36:06.805 CET Connection lost: The server closed the connection. ​

Logs cloud sql

2022-11-04T20:35:31.939856Z 271594 [Note] Aborted connection 271594 to db: 'xxx' user: 'xxx' host: 'cloudsqlproxy~xxx' (Got an error reading communication packets)

2022-11-04T20:35:31.940122Z 271565 [Note] Aborted connection 271565 to db: 'xxx' user: 'xxx' host: 'cloudsqlproxy~xxx' (Got an error reading communication packets)

2022-11-04T20:35:31.940151Z 271575 [Note] Aborted connection 271575 to db: 'xxx' user: 'xxx' host: 'cloudsqlproxy~xxx' (Got an error reading communication packets)

2022-11-04T20:35:31.940063Z 271574 [Note] Aborted connection 271574 to db: 'xxx' user: 'xxx' host: 'cloudsqlproxy~xxx' (Got an error reading communication packets)

[JGerman23 · Accepted solution]

My suggestion will be to create a PIT (Public Issue Tracker - Cloud Run), or please engage GCP Support if you're paying or if you're interested in starting to pay for a Support Package. Please be aware that from these 2 options, the second one is the fastest.

[JGerman23]

The issue you are getting might be related to different reasons.

• The server was too busy to allow any new connections, or only some of them. If this is the case, the connection attempt times out and produces this error.
• Numerous prior connections that should have been closed improperly are still resting inside MySQL until wait_timeout. The connection produces this error message after they time out in this manner.

With that in mind, there may be a brief outage of the database. Check the instance's monitoring for excessive load (such as high IOPS, disk activity, or stuttering heartbeats), especially if the connection failures arose quickly. Retry the connection if the load is excessive, preferably utilizing exponential backoff to stop your application from making too many connection requests at once.

There can be a buildup of long-standing, dormant connections. Run the show full processlist; on the database and look for connections with high "Time" or connections where "Command" is "Sleep" to inspect active connections.

Additionally, here is a Stackoverflow answer related to your question.

[Stijno24]

Thanks for your reply. The load on the servers is low at the moment it happens, what makes me suspect the cloud sql proxy is that we see these intermittent issues happen at the same time on different SQL instances. I dont see how i can investigate this without google admin access, so could you please assist?

[Stijno24]

@JGerman23 Any input ?

[Stijno24]

@JGerman23 Any input?

[JGerman23 · Accepted solution]

My suggestion will be to create a PIT (Public Issue Tracker - Cloud Run), or please engage GCP Support if you're paying or if you're interested in starting to pay for a Support Package. Please be aware that from these 2 options, the second one is the fastest.

[Stijno24]

OK i will open a issue tracker

[Stijno24 · Accepted solution]

Seeing the view count is nearing 5000 views, and having the issue again today, we still think its a broader issue.

Public issue tracker https://issuetracker.google.com/issues/260517774?pli=1 was denied and redirected to paid support, now going through account manager in gcp