DataStream connection to PostgreSQL using Private Ip getting time out error - Bigquery

[Abusalem-TL]

Hey everyone,

I'm having trouble setting up GCP connectivity for DataStream using a private IP to connect to PostgreSQL and BigQuery. I've created a Private connectivity config, but when I tried to create a connection profile, I kept getting a "timeout expired" error. I'm sure I've configured everything correctly. My PostgreSQL uses a private IP and I've entered the correct datasource details. Can someone please help me with this?

I'm attaching below my screenshots: 

Private Connectivity Config : 

[ms4446]

To troubleshoot the timeout issue when connecting Google Cloud DataStream to Cloud SQL Postgres using a private IP, you can follow these steps:

1. Private Services Access:

   • Ensure you've enabled Private Services Access for your VPC Network. This allows communication within your private network to Google-owned services.

2. VPC Network Peering:

   • Double-check that your VPC Network is correctly peered with the servicenetworking-googleapis-com network. This peering establishes the connection to the private IP of your Cloud SQL instance.

3. Firewall Rules:

   • Confirm that your firewall rules allow ingress traffic on the private IP of your Cloud SQL instance, specifically from the DataStream service.

4. Exported Routes:

   • In the VPC Network Peering settings, verify that the route containing the internal IP address of your Cloud SQL instance appears under the "Exported routes" tab.

Check Connection Profile Details:

1. Hostname/IP:

   • Make sure you're using the correct private IP address of your Cloud SQL Postgres instance in the connection profile.

2. Port:

   • Verify that the port number (usually 5432 for Postgres) is correct in the connection profile.

3. Credentials:

   • Ensure the username and password in the connection profile are accurate and have the required permissions to access your Cloud SQL database.

Cloud SQL Instance Configuration:

1. Private IP Configuration:

   • Confirm that your Cloud SQL Postgres instance is correctly configured with a private IP address.

2. Database Availability:

   • Ensure that your Cloud SQL Postgres instance is running and accessible within your private network. Try connecting to it directly using a tool like psql from a VM within your VPC.

3. Authorized Networks:

   • If applicable, check that the DataStream service IP ranges are added to the authorized networks for your Cloud SQL instance.

DataStream Configuration:

1. Private Connectivity:

   • In DataStream, verify that you have selected the correct private connectivity configuration when creating your stream.

Additional Tips:

1. Reverse Proxy (if applicable):

   • If you're using a reverse proxy, ensure it's correctly forwarding connections to your Cloud SQL instance's private IP address.

2. DNS Resolution (if applicable):

   • If you're using a private DNS zone, confirm that the private IP address of your Cloud SQL instance is correctly resolved.

3. Cloud SQL Auth Proxy (Optional):

   • Consider using the Cloud SQL Auth Proxy as an alternative way to establish secure connections to your Cloud SQL instance.

Example Firewall Rule:

• Direction: Ingress

• Action on match: Allow

• Targets: All instances in the network

• Source filter: IP ranges

• Source IP ranges: 35.191.0.0/16, 130.211.0.0/22

• Protocols and ports: tcp:5432

[edmariovalgueir]

---

@ms4446 wrote:

• • Hello! How are you? You can help-me, please? I was some days trying it to connect with same source and i get the same error. I have done with sucessufly all steps, but i have some doubts in these step:
    
    

    1. Exported Routes:

       • In the VPC Network Peering settings, verify that the route containing the internal IP address of your Cloud SQL instance appears under the "Exported routes" tab.

You can send images or detail it, please? Thank you so much.

---

---

@ms4446 wrote:

nsure that your Cloud SQ

---

[ms4446]

Here’s how to verify and configure the exported routes in the VPC Network Peering settings:

1. Access VPC Network Peering:
   • Go to the Google Cloud Console.
   • Navigate to the VPC Network section by selecting the hamburger menu (☰) > VPC network > VPC network peering.
2. Locate Your VPC Peering Connection:
   • Identify the VPC peering connection that links your VPC network to the servicenetworking-googleapis-com network.
   • Click on the VPC peering connection to view its details.
3. Check Exported Routes:
   • Within the peering connection details, look for the Exported routes tab.
   • Ensure that the route(s) including the internal IP address range of your Cloud SQL instance is listed here. The route should cover the subnet that contains your Cloud SQL private IP.

Example Steps for Viewing and Adding Exported Routes

1. View Exported Routes:
   • In the VPC network peering details page, select the Exported routes tab.
   • Review the list of exported routes to ensure the IP range of your Cloud SQL instance is included.
2. Add Exported Routes:
   • If the required routes are not listed, you may need to add them manually.
   • Go to VPC networks > Routes.
   • Click on Create route.
   • Fill in the necessary details such as:
     • Name: A unique name for the route.
     • Network: Select the VPC network you are working with.
     • Destination IP range: Enter the IP range that includes your Cloud SQL instance’s private IP.
     • Next hop: Select the appropriate next hop option, typically Default internet gateway or another VPC peering connection.
   • Save the route.

Example of Verifying and Adding a Route:

Verifying Routes:

1. Navigate to VPC Network Peering:
   • In the Google Cloud Console, go to VPC network peering.
2. Select Your Peering Connection:
   • Click on the peering connection name linking your VPC to the servicenetworking-googleapis-com.
3. Check Exported Routes:
   • Ensure the route that includes the IP range of your Cloud SQL instance is listed under Exported routes.

Adding a Route (if necessary):

1. Navigate to Routes:
   • In the Google Cloud Console, go to VPC networks > Routes.
2. Create a New Route:
   • Click on Create route.
   • Fill in the route details:
     • Name: my-cloudsql-route
     • Network: Your VPC network
     • Destination IP range: The IP range for your Cloud SQL instance, e.g., 10.0.0.0/16
     • Priority: 1000 (default is fine)
     • Next hop: Default internet gateway or VPC peering
3. Save the Route:
   • Click Create to save the new route.

[erosfdz]

Hi @Abusalem-TL , did you solve the connectivity problem? I'm having the same problem connecting from Datastream to AlloyDB? 

[Abusalem-TL]

Hi @erosfdz, 

I have resolved the issue. If you are experiencing the same problem, try to understand the VPC configuration. The SQL instance is located within the VPC, but the Datastream is not. Therefore, we need to use Compute Engine as a connector and connect using the Compute Engine's IP address.

[erosfdz · Accepted solution]

Thanks @Abusalem-TL 
Reading a little in the official documentation, find the same answer.
Thanks for your time to answer
https://cloud.google.com/datastream/docs/private-connectivity

[Abusalem-TL]

Yep Thanks.