Connection Refused without explanation, Google Proxy suddenly stops working. No error in log

[kensilverman]

Hello.  After yearrs of success, suddenly connection refused.  Restarted google Proxy with verbose,  no error messages.  Starts up ok.  I changed ulimit to 9000 (it only requests 8500) so no errors.  and no errors posted to the log when I get my connection refused error.  The creds file is correct.  Nothing is new or has changed.  Why suddenly this connection refused error?  It is not an access denied error.  I am accessing a database named global with a specific user that I added via phpmyadmin that has select permissions.  NTL,   what other permission should I suddenly need toi get mysqli-> command in php to work.  It always worked.,  Jsut suddenly stopped working?  What changed?  I did not change anything on my server.  the username and password appear correct. google cloud Proxy shows no error.

[ms4446]

Here  are some troubleshooting Steps for 'Connection Refused' Error in Google Cloud SQL

1. Verify Cloud SQL Instance Status and Configuration:

   • Instance Status: Ensure your instance is in the "Running" state and not exceeding its connection limits.
   • Authorized Networks: Confirm your IP is listed under "Authorized Networks." Adjust Public IP and Private IP settings as needed.
   • Project and Settings: Check if the instance has been moved or if settings have changed inadvertently.

2. Thoroughly Check Firewall Rules:

   • Cloud SQL Firewall: Confirm rules allow traffic from your IP on the correct port (usually 3306 for MySQL).
   • Local Firewall: Temporarily disable local firewall rules that might block outgoing traffic to Cloud SQL.

3. Inspect Network Connectivity and DNS:

   • Port Connectivity: Use telnet <Your-Cloud-SQL-Instance-IP> 3306 to test port connectivity.
   • DNS Issues: If DNS resolution problems occur, use the IP address directly in your connection string.

4. Deep Dive into User Credentials and Permissions:

   • Credentials: Double-check the accuracy of username and password.
   • Permissions: Verify the user has necessary permissions on the "global" database, especially if host specifications have changed.

5. Client-Side Considerations:

   • Library Updates: Check for compatibility issues with recent PHP or mysqli extension updates.
   • Network Changes: Review any recent network configuration changes on your server.

6. Seek Clues in Logs:

   • Cloud SQL Logs: Look for errors or warnings in Cloud SQL logs around the time of connection issues.
   • Server-Side Logs: Examine web server (e.g., Apache, Nginx) and PHP error logs, as well as system logs for network-related errors.

7. Cloud SQL Proxy and ulimit:

   • Proxy Status: Ensure Cloud SQL Proxy is up-to-date and functioning correctly.
   • ulimit: Verify sufficient file descriptors are available, though this is usually not directly related to connection refused errors.

8. Additional Tips:

   • SSL/TLS Connections: If using SSL/TLS, ensure certificates are valid and not expired.
   • Quota and Billing: Check for any quota limits or billing issues impacting the service.
   • Transient Issues: Consider the possibility of a temporary network issue or a service blip on the Cloud SQL side.

9. Contacting Support:

   • If these steps don't resolve the issue, reach out to Google Cloud SQL Support for further assistance.

[kensilverman]

1) instance is running via cloud_sql_proxy command with cert file specified to be sure (although I have never needed to specify the cert file in the past)  I run it in verbose now and direct output to error log - there is no error reported.  If I run a connectivity test via google console to my server where google proxy is located  it passes.  that ip address is authorized.  There have been NO changes.  It just suddenly stopped working.  Error log on google showed nothing.  Just a TLS warning for V1.  so I modified my sss.conf for only v1.2   (My  old Aws ec2 linux 2 does not support v1.3)  but that is not the issue.  STILL get connection refused errors and there are no firewall rules on my server.  the ip address IS authgorized as verfied in the console.  I have made no changes to the MYSQL user or password. 

Oddly, if I kill the mysql_proxy service, the connnectivity console test STILL passes!   I guess it does check google proxy service state?       Anyway, there is no reason I can find for this connection refusal when in php the mysql command is issued (with same user, password and database as I've used for months successfully.  and it is quite sad that no error log will report anything on it.   I updated my LetsEncrypt certificates like every 3 months, as in maybe that would effect it, but those are website certs.  I do not use any certs other than what google proxy automatically sets up.  Any other ideas here?

---

---

[kensilverman]

sorry to be clear, it is the php MYSQLI command generating the php exception that I catch and post to user with no log error reported anywhere. no firewall rules, no changes except ulimit.  I changed the ulimit because I was getting connection refused errors so I though the proxy startup warning might have been related.  Clearly it is not.

[kensilverman]

oh and my number of connctions as shown in console is <0.1K  (as in zero)   and the max allowed shows as 4000.    Although I did set my ulimit to 9000 to avoid the warnings I got with requests for 8500 upon startup.  That was the ONLY change I made.  Maybe I need to try again at 1024, the prior limit (by removing any ulimit directive in the relevant conf file) and ignore the proxy startup warning.

[ms4446]

Thank you for the clarification. Given this additional information, it seems the issue is more specifically related to the PHP mysqli command and its interaction with your Cloud SQL instance. Here's a revised approach considering your context:

1. Revert ulimit Changes:

   • Since changing the ulimit was the only recent change, consider reverting it back to the original value (1024) to test if this resolves the issue. The warning from the proxy startup might not be directly related to the connection refused errors.

2. PHP mysqli Command and Error Handling:

   • Ensure that the PHP mysqli command is correctly formatted and the parameters (hostname, username, password, database name) are accurate.
   • Enhance error handling in your PHP script to catch and log more detailed information about the mysqli error. This can provide more insights into the nature of the connection issue.

3. Check PHP Environment:

   • Verify the PHP version and mysqli extension version. An update or mismatch in versions can sometimes cause unexpected issues.
   • Check for any recent updates or changes in your PHP environment that might affect database connections.

4. Database Connection Parameters:

   • Double-check the database connection parameters in your PHP script. Ensure that the host, port, username, password, and database name are correct and match the configuration in Google Cloud SQL.

5. Test with a Simple PHP Script:

   • Create a simple PHP script that only attempts to connect to the database using mysqli. This can help isolate the issue from other parts of your application.

6. Cloud SQL Logs and Metrics:

   • Review the logs and metrics in the Google Cloud Console for any anomalies or errors that correspond to your connection attempts.

7. Network and Connectivity Checks:

   • Although you mentioned no firewall rules, it's still worth checking if there are any network issues or changes that could be affecting connectivity.

[kensilverman]

Apparently my unix sockets must have somehow switched off like magic, so the php  mysqli command had to be changed from "localhost" to "127.0.0.1"  to suport tcp for the google mysql proxy.  Kind of ridiculous.  Not sure how or why the socket ability turned off.  All I know is that fixed it, when for years localhost at 3307 was working fine.   And my php error checking even with trace did not report this problem.  How do I check for unix sockets so that I can get that working again and any ideas how it switched off?  (I am just assuming it did.)