CloudSQL private connectivity issue

[PetroK]

Hi folks, I hope somebody can advise on the following issue.  I've enabled private service access for Cloud SQL, allocated CIDR range, VPC peerings were created as well automatically.  Also created site-to-site VPN between AWS and GCP VPC's.   Now I need to connect from EC2 instance in AWS to CloudSQL for MySQL DB.  I've followed instructions here https://cloud.google.com/sql/docs/mysql/configure-private-ip#vpn and updated 3 peering connections with enable Export custom routes and also enabled custom route advertisement in Cloud Router settings. 

For cloudsql-postgres-googleapis-com peering connection routes propagated fine and I can telnet to DB, but for cloudsql-mysql-googleapis-com some routes (including route to AWS) produce error " rejected by peer configuration" and I can't telnet to MySQL databases.   

I don't know if this is some bug on GCP side or how to troubleshoot why routes were rejected particularly with mysql peering, but not with Postgres.

[ms4446]

There are a few things you can check to troubleshoot why your routes are being rejected by the peer configuration for the cloudsql-mysql-googleapis-com peering connection:

1. Make sure that the VPC peering connections between AWS and GCP are both active and configured correctly. This includes enabling Export custom routesand Import custom routes for both peering connections.
2. Make sure that the custom routes are enabled for the cloudsql-mysql-googleapis-com peering connection.
3. Make sure that the custom routes are advertised in the Cloud Router settings.
4. Make sure that the CIDR range for the AWS VPC is not overlappingwith any other CIDR ranges in use.
5. Make sure that the EC2 instance in AWS is in a subnet that is peered with the GCP VPC.
6. Check the Cloud Router logs to see if there are any errors related to the custom routes.

Advanced Troubleshooting

1. Try disabling and then re-enabling the custom routes.
2. Try deleting and then re-creating the custom routes.
3. Try connecting to the Cloud SQL instance from a different EC2 instance in AWS.
4. Contact GCP support for assistance.

Alternative Solution

If you are unable to resolve the issue using the above troubleshooting steps, you can try using a Cloud SQL proxy. A Cloud SQL proxy is a secure and reliable way to connect to a Cloud SQL instance from any network.

To use a Cloud SQL proxy, you will need to install the Cloud SQL proxy client on the EC2 instance in AWS. Once the client is installed, you can start the proxy and connect to the Cloud SQL instance using the following command:

./cloud_sql_proxy -instances=YOUR_PROJECT_ID:YOUR_REGION:YOUR_INSTANCE_NAME=tcp:3306

Once the proxy is running, you can connect to the Cloud SQL instance using the following command:

mysql -h 127.0.0.1 -P 3306 -u YOUR_USERNAME

The Cloud SQL proxy will encrypt all traffic between the EC2 instance and the Cloud SQL instance.

Important: Make sure that the CIDR range for the AWS VPC is not overlapping with any other CIDR ranges in use. Overlapping CIDRs are a common source of issues in such setups.

[PetroK]

Thanks for you reply @ms4446 , I will try to check for overlapping ranges, but I found it really strange that it works cloudsql-postgres-googleapis-com, but not for cloudsql-mysql-googleapis-com.   And unfortuinately I can't use Cloud proxy in this particular case, as I need to enabled CDC replication to RDS instance, so I need direct connectivity, as I won't be able to install proxy on RDS instance. 

[ms4446]

It is indeed unusual that the VPC peering connection works for cloudsql-postgres-googleapis-com but not for cloudsql-mysql-googleapis-com. This discrepancy could arise from a specific configuration difference between the two or some unusual behavior in the Cloud SQL service for MySQL.

If you've ensured there are no overlapping CIDRs and all other settings are correctly configured, I'd recommend reaching out to GCP support. They'll be best positioned to dive deeper into the issue and offer a resolution.

In the interim, for enabling CDC replication to your RDS instance, you might consider a few workarounds:

• Third-party CDC Services: There are several third-party CDC services available that might fit your needs. However, integrating a new service can come with its own set of challenges and potential costs.

• Cloud SQL Proxy on Separate EC2: You could set up a Cloud SQL proxy on a standalone EC2 instance and configure your CDC replication to connect through this proxy. It's worth noting that introducing this intermediary might add some latency, which could be significant depending on your real-time replication requirements.

I recognize that these solutions might not be ideal, but they could serve as temporary measures until the primary issue is addressed.