GKE private cluster (autopilot) has no VPC peering after startup

[pwolfe]

Today I've been trying to spin up a new cluster in terraform and have been unable to add a peering route between my network and the control plane. When I check the private cluster config I get the following:

gcloud container clusters describe <cluster> --region=europe-west4 --format='value(privateClusterConfig)'
enablePrivateEndpoint=True;enablePrivateNodes=True;masterGlobalAccessConfig={'enabled': True};privateEndpoint=10.164.15.201;publicEndpoint=35.204.136.72

Note there is no peeringName in the config and no master cidr block is automatically assigned. This used to happen automatically. My terraform looks like:

resource "google_container_cluster" "wgs_cluster" {
  project          = var.project_id
  name             = var.cluster_name
  location         = "europe-west4"
  enable_autopilot = true
  network          = "default"
  subnetwork       = "default"
  ip_allocation_policy {
  }
  private_cluster_config {
    enable_private_endpoint = true
    enable_private_nodes    = true
    master_global_access_config {
      enabled = true
    }
  }
  master_authorized_networks_config {
    cidr_blocks {
      cidr_block   = "192.168.0.0/16"
      display_name = "vpc"
    }
  }

  lifecycle {
    ignore_changes = [master_auth, network, dns_config, gateway_api_config]
  }
}

 Any ideas?

[garisingh]

Check the release notes from March 11.  Starting with GKE 1.29.3, private clusters now use PSC (which has been the case for public clusters for a while now) rather than VPC peering.  This blog from Dec 2022 explains the new control plane networking (it was about public clusters, but now this applies to new private cluster 1.29.3 and later as well).

[m4Lim3]

Thanks. Is there a way to provision 1.29 with VPC peering ? This has broken our tf automation.

[garisingh · Accepted solution]

There is not.  You'd need to use 1.28 or earlier.

I'd like to understand how this breaks your automation ... are you checking for something after the fact?   Generally speaking, this should be more or less a transparent change.

[m4Lim3]

Thanks. Correct. GKE peering update is part of our pipelined tf run (importing/exporting custom routes). Errors out now and as peering property on gke cluster resource is ''.

[pwolfe]

Thanks @garisingh . I have a similar issue to @m4Lim3 . We use the tutorial here to connect our cloud build worker pools to our private clusters: https://cloud.google.com/build/docs/private-pools/accessing-private-gke-clusters-with-cloud-build-pr.... That tutorial requires a peering route to be made using the peering name (also to import export custom routes). Is there a workaround with a  1.29 cluster, I'd rather not roll back?

Also, that documentation and this one could use an update https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#vpc_peering_reuse. 

[garisingh]

Thanks for pointing this out.  We do have a new feature coming in GKE which will remove the need to do this at all coming in 4Q, but we'll take a look at this in the meantime to figure out what can be done with the current PSC architecture.  I actually think the GKE control plane endpoint will actually still be available even when using the architecture in the tutorial above since there's a VPN tunnel in place so the control plane endpoint should actually still be available.

Also, would you mind using the "Feedback" button on the tutorial page you linked to file the issue?  You can also ask them to reach out to me as well when you submit the feedback.

[pwolfe]

I've sent Feedback on the tutorial. I have tried to just remove the peering route update step, however my build cannot hit the control plane (times out). I'll revert to v1.28 for now as I think if I just bootstrap the cluster with that version I'll get the peering and can upgrade it after.

But please let me know if you find a path to use PSC without any intervention. Thanks!

[garisingh]

Using https://cloud.google.com/build/docs/private-pools/accessing-private-gke-clusters-with-cloud-build-pr... with a couple modifications, I'm able to use this pattern to access the new PSC-based private clusters (created with 1.29.3 and later) from a Cloud Build private pool.

1) Under 

https://cloud.google.com/build/docs/private-pools/accessing-private-gke-clusters-with-cloud-build-pr... you don't (and can't) do steps 2 and 3

2) You'll need to create a firewall rule allowing access to the CLUSTER_CONTROL_PLANE_CIDR range from the PRIVATE_POOL_NETWORK/PRIVATE_POOL_PREFIX

[ylakkaraju-st1]

@garisingh Could you please state what changes were made from the tutorial?

with a couple modifications, I'm able to use this pattern to access the new PSC-based private clusters (created with 1.29.3 and later) from a Cloud Build private pool.

---

1. I was anyway not able to perform the steps in this section of the guide since the version of our Autopilot Private Cluster is v1.30.2-gke.1587003.

2. I have created a FW rule (VPC firewall rules) in the direction of Ingress with below config:

Direction: Ingress

Source filters: PRIVATE_POOL_NETWORK/PRIVATE_POOL_PREFIX

Destination filters: CLUSTER_CONTROL_PLANE_CIDR

What am I still missing? 

[garisingh]

@pwolfe  - you might want to check out the new DSN-based access for the control plane:

https://cloud.google.com/kubernetes-engine/docs/concepts/network-isolation#dns-based_endpoint

[m4Lim3]

Same here - private cluster, shared vpc, private cloud build pools, needs peering to gke control plane and vpn gw architecture so you can hit control plane from cb to deploy k8s manifests.  Only way i found to retain current set up is to provision gke @ v1.28.