Cloud Run connection to Cloud SQL times out occassionally

[nifk01]

Hello everyone!

We've been deploying some Go 1.17 servers on Cloud Run and they're connecting to a Cloud SQL (Postgres) instance over a public IP (using the Cloud SQL proxy).

Occassionally, our users notice HTTP 500 errors, due to db connection issues. At the same time, if I view all of our GCP logs, there are some error lines from the Cloud Run revisions like this:

Cloud SQL connection failed. Please see https://cloud.google.com/sql/docs/mysql/connect-run for additional details: connection to Cloud SQL instance at <public-db-ip>:3307 failed: timed out after 10s

There are also some like this:

Please see https://cloud.google.com/sql/docs/mysql/connect-run for additional details: read tcp 169.254.8.2:63521: connection reset by peer

A few minutes before  and after these errors, requests are being processed normally by our containers. I've searched the database logs and that are only normal info messages, nothing suspicious. The database is also under-utilized (CPU, memory, disk I/O). And we have no big request counts in our containers.

The Postgres configuration is as follows:

=> SELECT name, setting FROM pg_settings WHERE name LIKE 'tcp_keepalives_%';
name | setting
-------------------------+---------
tcp_keepalives_count | 5
tcp_keepalives_idle | 60
tcp_keepalives_interval | 60
(3 rows)

Any ideas on how this could be investigated further?

Thanks in advance!

[nifk01]

I've just enabled `log_connections` in Postgres and waiting for this to occur again. I'll follow up when it does.

[silvers]

Started seeing this recently too. We are python/django. Also CloudSQL, postgresql, public IP with cloud sql proxy.

Cloud SQL connection failed. Please see https://cloud.google.com/sql/docs/mysql/connect-run for additional details: connection to Cloud SQL instance at x.x.x.x:3307 failed: timed out after 10s

[nifk01]

It stands out to me that the error talks about MySQL (port 3307) instead of Postgres (port 5432). Both in my original and @silvers error.

@silvers FYI I've opened a ticket on the Issue Tracker here. You might want to chime in.