Whitelist IP Address for Vertex Pipelines in a VPC Network with Cloud NAT

[k00kI03]

Hi everyone,

I'm running Vertex Pipelines in a VPC network with Cloud NAT attached. I've configured firewall rules on my target servers to whitelist the IP addresses of my VPC network(NAT). However, I'm encountering issues because the Vertex Pipelines instances seem to be using different public IP addresses each time they run.

I assumed that since the pipelines are running in the VPC and using Cloud NAT, they would use the same public IP address for egress traffic going to other networks. However, I'm seeing a variety of public IP addresses from different subnets within the europe-west2 region (where pipelines are running).

I'm wondering if there's a way to ensure that Vertex Pipelines instances use a consistent public IP address for egress/ingress traffic, or if I need to adjust my firewall rules to allow traffic from a wider range of IP addresses.

Any insights or suggestions would be greatly appreciated.

Thanks

[diannemcm]

Hi @k00kI03,

Welcome to Google Cloud Community!

You may refer to this documentation from @Chris Willis, the guide outlines a workaround for users who need to guarantee that their Pipelines traffic originates from specific IP addresses.

This architecture visualizes the setup of a NAT instance with two interfaces, one in your VPC peered network and the other one in an outbound network. A static method in order to allow your device to communicate with googleapis.com.

Also for additional references, you can check this blog post from @Bernie Ongewe:

Vertex AI Pipelines and jobs use the Service Networking API to configure their networking. As such, they will run in a Google-owned VPC network in tenant project that is peered to the network in your project. This peering is subject to the following standard VPC Peering networking constraints.

I hope the above information is helpful.

[dh-raufbhat]

Hello, Thanks for the detailed and step by step  explanation.

I was able to setup everything till  "Route traffic only to the specified IPs via the default gateway of ens4"

Since my project doesn't have a default network due to organisational settings. I run into an error when I execute the below commands.

# Example: this will route any packets destined for the IP of ifconfig.me or the two major AWS IP ranges to the nat-instance
gcloud compute routes create manual-34-to-nat-instance --destination-range=34.0.0.0/8 --next-hop-instance=nat-instance --priority=100
gcloud compute routes create manual-52-to-nat-instance --destination-range=52.0.0.0/8 --next-hop-instance=nat-instance --priority=100
gcloud compute routes create manual-54-to-nat-instance --destination-range=54.0.0.0/8 --next-hop-instance=nat-instance --priority=100

Error:

ERROR: (gcloud.compute.routes.create) Could not fetch resource:
 - The resource 'projects/[project_id]/global/networks/default' was not found

1. What do you suggest here?
2. Does this way of VPC peering impact all the service in google tenant project or can it be customised only for Vertex AI?