Unable to provision a GKE auto pilot cluster

[Dg03cloud]

I'm trying to create a GKE auto pilot cluster through terraform. For this I have created a shared host service project with a shared network. Followed this google reference to create a gke cluster resource. 

For the terraform managed service_account to create a cluster, I have enabled these list of apis (roles/container.admin & roles/compute.networkViewer).  

https://github.com/terraform-google-modules/terraform-google-kubernetes-engine/tree/v30.2.0/modules/...

I got the following exception while provisioning this gke resource. 

Error: googleapi: Error 403: Google Compute Engine: Required 'compute.subnetworks.get' permission
Details:
│ [
│ {
│ "@type": "type.googleapis.com/google.rpc.RequestInfo",
│ "requestId": "0x5c6ab2d64f295d33"
│ }
│ ]
│ , forbidden 

Any helpful thoughts on this issue would be great. 

[DamianS]

Hello @Dg03cloud  ,Welcome on Google Cloud Community.

Are you able to grant little higher permissions for service account, than Compute Network Viewer ? Like for example, Compute Network User for tests ? 

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost

[Dg03cloud]

Hi @DamianS, 

Appreciate for your prompt reply.  I have tried with this permission 'roles/compute.networkUser'. Still I'm seeing the same exception as below. 

Error: googleapi: Error 403: Google Compute Engine: Required 'compute.subnetworks.get' permission
Details:
│ [
│ {
│ "@type": "type.googleapis.com/google.rpc.RequestInfo",
│ }
│ ]
│ , forbidden.

[DamianS]

Hmm, 
Are you able to check logs via Logs Explorer ? Also, are you trying to create this gke cluster under the project, where you've configures service account ?  Are you able to describe your setup ? 
--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost

[Dg03cloud]

Yes, I have checked the logs as well. Using the correct project_Id at terraform config, and the associated service account is granted with a necessary permissions too. I haven’t tried with the compute network admin role yet. But I believe the project  service_account is having a compute network user role which is pretty much enough to read the subnet networks from the shared host project.

[DamianS]

Hmm, 
If I understood correctly, you are using shared VPC ? So, you are trying to deploy GKE at projectA, but VPC is shared from ProjectB ? If yes, did you've tried to add your SA account with those permissions on both projects? 

cheers,
DamianS

[Dg03cloud]

@DamianS Appreciate your prompt reply.

So, in the project-A (where I'm going deploy a gke cluster) created own service account (EX: name GKEService-Account) and assign this role 'Kubernetes Engine Cluster Admin'. Then in the project-B (which is a shared host project) for this 'GKEService-Account' added with this role 'Compute Network Viewer'. And I added service_account =  GKEService-Account.service_Account.com in the gke terraform child module as well. Once I run the terraform apply, I got this error "required: compute.subnetworks.get" Let me know if I missed anything here.

[DamianS · Accepted solution]

@Dg03cloud You're welcome 🙂 I'm always happy to help. 

Enable :

Kubernetes Engine API for both projects
Compute Engine AP for both projects
cloudresourcemanager.googleapis.com for both projects

Ensure, that shared VPC have "Kubernetes Engine access" on Enabled

my setup:

module "gke" {
  source                     = "terraform-google-modules/kubernetes-engine/google//modules/beta-autopilot-private-cluster"
  project_id                 = "PROJECT_A"
  network_project_id         = "Project-B"
  name                       = "gke-test-1"
  region                     = "us-central1"
  zones                      = ["us-central1-a", "us-central1-b", "us-central1-f"]
  network                    = "vpc-01"
  subnetwork                 = "us-central1-01"
  ip_range_pods              = "us-central1-01-gke-01-pods-2"
  ip_range_services          = "us-central1-01-gke-01-services-2"
  horizontal_pod_autoscaling = true
  #filestore_csi_driver       = false
  enable_private_endpoint = false
  enable_private_nodes    = false
  master_ipv4_cidr_block  = "10.0.0.0/28"
  service_account         = "gkeservice-account@PROJECT-A.iam.gserviceaccount.com"

}

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost

[DamianS]

Works for me 

[Dg03cloud]

@DamianS Thanks for sharing. 

Just to confirm, my project-A and project-B are already enabled with these list of apis "Kubernetes Engine API", "Compute Engine API",  "cloudresourcemanager.googleapis.com". 

And as suggested, I also tried to enable "Kubernetes Engine access" api on the shared-host projectB, but I couldn't able to find that in the given permission/roles list to enable. Any thoughts on this?

And my gke-auto-cluster terraform module configuration is also quite similar. 

[DamianS]

You should see this under Shared VPC -> Attached projects -> ProjectA. 

[Dg03cloud]

@DamianS Appreciate your time. I' able to fix this issue at shared vpc attached project. Reattach my project to the shared host project with this role 'kubernetes engine access', then the cluster got created with out any issues. Thanks.

[DamianS]

@Dg03cloud ,

Brilliant! Please do not forget to pick solution for your issue and close this topic 🙂 

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost

[DamianS]

@Dg03cloud 
Additionally, I found this documentation about clusters and shared vpc. Maybe this will somehow help

https://cloud.google.com/kubernetes-engine/docs/how-to/cluster-shared-vpc#gcloud_3

[DamianS]

Additionally I found one issue :

damian_sztankowski@cloudshell:~/gke-auto (webaap-wordpress-load)$ terraform validate
╷
│ Error: Unsupported argument
│ 
│   on main.tf line 21, in module "gke":
│   21:   filestore_csi_driver       = false
│ 
│ An argument named "filestore_csi_driver" is not expected here.

--
cheers,
DamianS
LinkedIn medium.com Cloudskillsboost