Reuse address space for multiple GKE clusters anywhere within GCP

[mandadhi]

Reuse address space for multiple GKE clusters anywhere within GCP - featuring NCC, PSC Interfaces, and Private NAT

Kubernetes distributions require a lot of IP addresses for pods, services and control plane endpoints; Google Kubernetes Engine (GKE) is no exception to this. As your network gets complex with hybrid cloud strategies you may find it difficult to both allocate, manage and scale your network. The goal of this blog is to present a GKE architecture that allows reusing the same address space for many clusters. The clusters can be in any project including different orgs in Google Cloud Platform (GCP).

Architecture

 “Inbound” diagram"                                                                     “Outbound” diagram" 

Deployment

The Terraform code for the architecture outlined in this document is available in the associated GitHub repo.

To get started with implementing this solution, follow these steps in the README.md file:

1. Create the NCC hub.
2. Update the following variables in terraform.tfvars:
1. ncc_hub_project_id
2. ncc_hub_name
3. network_name
4. gke spokes 
3. Run terraform apply.

Connectivity for applications

Connectivity to services is outlined in the following two sections: inbound and outbound.  “Inbound”, in the context of this blog, is for enterprise applications to talk to the applications deployed in the island GKE clusters. “Outbound”, in the context of this blog, is for GKE services to talk to the enterprise applications in the primary network. The Primary network in this example is your enterprise VPC and could be connected to your on-prem network.

Network connectivity center for outbound

As shown in the rightmost diagram above, the “Outbound” network path for services will use Network Connectivity Center (NCC). In this design, all VPC spokes containing the island GKE clusters will be connected to a single NCC hub with outbound spoke filters for all the CIDRs used by the clusters. This ensures that the address space can be reused across multiple clusters.  GCP Private NAT will be used for “outbound” connectivity from the GKE island spokes to the “primary” VPC. The Private NAT subnets used by the Private NAT Gateway are the only IP space which need to be unique across the spokes.  This is required in order for NCC to route traffic back to the proper island cluster. These Private NAT subnets will be the only IP space exported from the NCC spoke into the NCC hub. Your GKE workloads can talk to the enterprise applications directly since the enterprise network traffic is routable from the island networks. You need to add ingress Cloud Firewall rules with the source of the GKE island spoke Private NAT IP ranges.  These Cloud Firewall rules should be associated with the “primary” VPC targeting the resources running the enterprise applications.

At the time of writing, “NCC Export Include Filters is Private Preview”.  Once Generally Available, Export Include Filters can help simplify the configuration.

Network connectivity center for inbound

Network connectivity for “Inbound” (traffic initiated from the primary network to your GKE cluster) works differently than “outbound” connectivity.  In the case where the destination VPCs have the overlapping IP space, using NCC with Private NAT is not a viable solution. Cloud NAT (both public and private) provides source address and port translations for egress traffic. In this case, we need a destination based NAT solution since it is the destination addresses that overlap across the GKE clusters.  Cloud NAT does not offer destination address translation.  Rather than using NCC to route traffic between the VPCs, “inbound” connectivity uses PSC Interfaces, connecting each island GKE VPC to the Primary VPC with a dedicated Router VM.  Using PSC Interfaces allow the GKE VPCs to exist in different Google Cloud projects than your “primary” VPC.  You will need to reserve an IP address space in the primary network for the number of services you want to talk to on the island clusters. For example, if you need to expose 10 IPs in a GKE cluster, assign a /28 (16 addresses) in the primary network.  The Router VM is configured to destination NAT traffic from the primary network to the island network for the configured range using the iptables NETMAP target or action. NETMAP performs a 1:1 address translation for addresses inside a defined subnet to addresses in a similar size subnet. In the provided Terraform code, this can be configured per spoke using variable spoke_nat_subnet. This needs to be unique in your primary network. This makes the selected services natively routable from the primary network. The following diagram shows the network path from the primary network to an ILB of an application deployed in the island cluster.

In this example, “ens5” is a PSC Interface created in the island project and allows the primary project to assign the interface to the respective Router VM.  The Router VM will perform a static 1:1 nat from [10.244.0.34 to 100.64.84.2] and from [10.244.0.35 to 100.64.84.3].  While performing a 1:1 destination NAT, the Router VM will also perform a Source NAT (SNAT), masquerading traffic from its ens5 interface.  This will ensure that return traffic is routed back through the Router VM.  All of the Router’s NAT configuration is programmed within the Terraform code.  There are also static custom routes programmed in the “primary” networking pointing the NATed network to its respective Router VM.  The programming of these routes is also managed by code.  The network admin may need to configure Cloud Firewall rules targeting the Router VMs are required in order for the enterprise applications to reach the workloads on the specific ports used by the ILBs.

When to use this architecture

1. If you’d like to use the same address space for multiple GKE clusters.
2. You want to simplify network planning for GKE clusters in multiple GCP organizations.

Design considerations

1. For additional redundancy, the Router VMs for a particular spoke can be scaled up and included in an instance group.  The instance group can then be added as a backend of an internal network passthrough load balancer (ILB) and used as a next hop in the custom static routes in the primary VPC.  Because the Router VMs are SNATing all outbound traffic, the corresponding ILBs in the GKE VPCs are not required for return traffic.
2. To talk to services on-prem from GKE island networks, you’d need to include the ranges when adding the primary network as the NCC spoke.  Future features of NCC will allow this dynamically, however it is outside the scope of this blog.
3. Using the Router VM for “outbound” connectivity rather than NCC was considered.  However, it proved difficult to consistently deploy in code since the IP address used for the router appliance PSC interface could not be defined.  In addition, using NCC took some load off the router appliance and allowed for significant scale.
4. Using NCC allows for significant scale and additional connectivity options as new NCC features become available.
1. NCC would allow for 250 total VPC spokes, and 1000 Subnets routes in the NCC hub. 
5. Private NAT supports only TCP and UDP connections.
6. Additional Cloud Firewall rules may be required to permit the desired traffic flow. Here are some generic examples of FW rules that may be needed:
1. Inbound pattern rules
1. Primary VPC: Ingress Allow rule with the source of the clients, targeting the respective Router VMs.
2. GKE VPC: Ingress allow rules with source IP address of Router VM's egress interfaces
2. Outbound pattern rules:
1. Primary VPC: Ingress Allow rule with the source of the Private NAT IP space of the respective GKE VPC, targeting resources in the primary VPC.
7. Options to allow reachability to the GKE Master IPs from the “primary” VPC would require slight networking changes leveraging additional NETMAP configurations or PSC Propagation, through NCC (currently in Public Preview).
8. Consider using a NCC Star preset topology, where the primary network is an inner spoke and the GKE VPCs are outer spokes, if you would like to prevent the GKE islands from exporting their Private Subnet (or any subnet) to other GKE islands regardless of Export Filters. At the writing of this blog, Star topology support is Public Preview and therefore was not used in the deployment example. Star topology only supports VPC spokes.

Network planning used in this architecture

The following CIDRs are examples used in this architecture and may need to be adjusted to meet your networking and scale requirements.

VPC	IP Space	Function or Description	Reused across islands
Primary	10.128.0.0/24	Primary VPC subnet	NO
Primary	10.244.0.X/28	1:1 NAT for GKE ILB subnet	NO
GKE spokes	100.64.0.0/19	Primary subnet used by the nodes	Yes
GKE spokes	100.64.32.0/19	Pods secondary range	Yes
GKE spokes	100.64.64.0/20	Services secondary range	Yes
GKE spokes	100.64.96.X/28	Subnet for GKE master	Yes
GKE spokes	100.64.83.0/24	Regional managed proxy subnet	Yes
GKE spokes	100.64.84.0/28	Subnet for ILBs / GKE gateways	Yes
GKE spokes	100.64.87.0/29	Subnet for PSC Interface	Yes
GKE spokes	100.65.X.0/24	Private NAT subnet	NO

References

1. About Private Service Connect interfaces | VPC | Google Cloud.
2. Network Connectivity Center documentation | Google Cloud.
3. Private NAT | Google Cloud.
4. Gateway | Google Kubernetes Engine (GKE).