Cloud Run can't connect to MySQL after Google Cloud SQL Maintenance

[KalleDelorean]

We are seeing a super weird issue regarding MySQL server connectivity from Cloud Run.

Our setup is two separate PHP applications in Cloud Run (one Wordpress, one Laravel).

Today, Google performed maintenance on the SQL server. After the maintenance was performed, the cloud run applications could not connect anymore. There have been no changes to code/env variables or anything.

The Laravel app had somewhat verbose logging and indicated that the connection resulted in access denied, so at that point it sounds more like an auth issue than connectivity.

Checking the Cloud SQL server MySQL error logs shows this to be the case as well:

`INFO 2025-04-22T07:01:45.444819Z 2025-04-22T07:01:45.444524Z 686 [Note] [MY-010926] [Server] Access denied for user 'redacteduser'@'cloudsqlproxy~34.34.30.5' (using password: YES)`

For every attempt to use the app, another log entry is produced.

We are using "standard" MySQL authentication and users. In other words, not IAM users.

This is the really weird part:

We can "resurrect" connectivity in one of the following ways from our testing:
- Connect with the affected MySQL user with the exact same credentials, using GCP console "Cloud SQL Studio".
- Initiate a connection from our local computers, for example set up a simple PHP script that connects.
- Deploy a new Cloud Run revision, with NO changes to code/environment variables.

After this, the apps work fine, without redeploying or making any changes whatsoever!

This is not intuitive at all, so I'll emphasize:
The mere act of connecting from somewhere else seems to "flush" some state somewhere, and from that point on connections are accepted.

The real problem from this is that maintenance may be redirected to "off-hours" and not disturb us/our clients, but it puts the apps in a state of not working, until someone makes a connection manually from somewhere. 

Any tips on how to troubleshoot this?

[mcbsalceda]

Hi @KalleDelorean,

Welcome to Google Cloud Community!

It sounds like the issue could be related to the state of the Cloud SQL connection or a temporary hiccup caused by the maintenance, especially if you're seeing "access denied" messages despite no changes in code or environment variables. One potential cause is that after maintenance, Cloud SQL might have had issues properly recognizing or refreshing the connection state, which could explain why connecting externally (from your local machine or Cloud SQL Studio) or redeploying helps "reset" things. I suggest checking the Cloud SQL troubleshooting guide for debugging connection issues, as it can help identify issues like authentication mismatches, firewall rules, or proxy issues.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[KalleDelorean · Accepted solution]

Hello @mcbsalceda ,

This is not an issue regarding my connection setup, it turns out it is only happening on MySQL 8.4, see this thread:

https://www.googlecloudcommunity.com/gc/Databases/Cloud-SQL-DB-access-denied-unless-also-logged-in-t...

Important notes regarding the behavior of GCP regarding this issue:

1. If you create a new MySQL user, it will NOT be able to connect over Cloud SQL Proxy,. We can regard this user as "dead" currently.
2. If you initiate a connection via Cloud SQL studio or similar means (custom connection script to the server), you will essentially "revive" the user so it can now connect via Cloud Run for example.
3. If you run the SQL command "FLUSH PRIVILEGES;" on the server, it will make ALL users "dead". These can then be revived individually by doing what's described in step 2.
4. If GCP performs maintenance on the server, this is EQUIVALENT to running FLUSH PRIVILEGES.

What is the important takeaway here?

Any GCP projects using MySQL 8.4 are ticking timebombs that will detonate at the time of maintenance being done.

DO NOT USE MySQL 8.4!!! I think GCP should hide this option for now until a solution has been implemented.

@mcbsalceda Since you are staff, could you advice me on what to do from here? Is there someone I should email about this?

I can confirm this is an issue that is quite serious, but I cannot persuade my managers to buy an expensive Google Support subscription to simply report what is essentially an error on Googles end (we users cannot mitigate this error).

Should I create a new thread?

[zlucas]

I have had exactly the same issue and have been pulling my hair out over it.

I can also confirm that after the maintenance update tonight that connections would no longer work from all apps to the Cloud SQL database. To solve the problem, I created a new password - however at that point it still did not work, and I had to login with the new password before the app would connect successfully via cloudproxy.