when deleting gke with oidc it not deleting firewall rules and the gke-oidc-envoy load balancer

[yossig-runai]

Hi, we are using terraform to deploy and destroy envs, we use very old provider and every thing work fine. but when we are updating the provider to the latest when we are deleting the GKE  it not deleting firewall rules and the gke-oidc-envoy load balancer. so we can't delete the vpc.
example error:

Error: Error waiting for Deleting Network: The network resource 'projects/run-ai-lab/global/networks/yossi-gke-n55-network' is already being used by 'projects/run-ai-lab/regions/us-east4/forwardingRules/a1172f6b41e3b434c9f1bea8f90c9654'

same issue also when creating with gcloud

[nmagcalengjr]

Hi @yossig-runai,

Welcome to Google Cloud Community!

It sounds like you're experiencing an issue with Kubernetes Engine (GKE) cleanup when using OpenID Connect (OIDC) authentication. Google Cloud sometimes takes time to fully delete network resources, causing a dependency issue.

It seems that the new Terraform provider version isn’t properly managing dependencies during environment teardown. When deleting a Google Kubernetes Engine (GKE) cluster with OIDC enabled, you must also remove related resources such as firewall rules and the gke-oidc-envoy load balancer. The error indicates that the VPC remains in use because the forwarding rule, along with other network resources, wasn’t deleted before Terraform tried to remove the VPC.

Here’s how you can clean them up manually:

1. Identify and Delete the Firewall Rules - GKE with OIDC creates firewall rules, typically prefixed with gke-oidc-. (These don’t always get removed when you delete the cluster)

• List Firewall Rules - Run the following command to find any remaining GKE-related firewall rules:

gcloud compute firewall-rules list --filter="name:gke-oidc-*"

• Delete the Firewall Rules - Once identified, delete them manually: (Replace FIREWALL_RULE_NAME with the actual rule name)

gcloud compute firewall-rules delete FIREWALL_RULE_NAME

2. Identify and Delete the gke-oidc-envoy Load Balancer - The gke-oidc-envoy component creates a load balancer that sometimes isn’t cleaned up.

• List Load Balancers

• Check the forwarding rules:

gcloud compute forwarding-rules list --filter="name:gke-oidc-envoy-*"

• Check the backend services:

gcloud compute backend-services list --filter="name:gke-oidc-envoy-*"

• Delete Load Balancer Components

• Delete the forwarding rule:
  gcloud compute forwarding-rules delete FORWARDING_RULE_NAME
• Delete the target proxy:
  gcloud compute target-http-proxies delete TARGET_PROXY_NAME
• Delete the URL map:
  gcloud compute url-maps delete URL_MAP_NAME
• Delete the backend service:
  gcloud compute backend-services delete BACKEND_SERVICE_NAME
• Delete the health check (if it exists):
  gcloud compute health-checks delete HEALTH_CHECK_NAME

3. Check for Orphaned Network Resources - You can check for unused IP addresses and delete them:

• List Orphaned IP Addresses

gcloud compute addresses list --filter="status:RESERVED"

• Delete Unused IPs

gcloud compute addresses delete IP_NAME

4. Verify and Clean Up Any Remaining Resources - Finally, run:

gcloud compute resources list --filter="name:gke-oidc-*"

If you encounter the same issue when creating a GKE cluster using gcloud, it indicates that the problem is not specific to Terraform but is instead related to how GKE with OIDC provisions and manages resource cleanup.

Here are some steps you can take:

1. Ensure Dependencies Are Handled Correctly in Terraform

- With the older provider, Terraform might have been automatically managing dependencies differently. The new provider might need explicit dependencies.

- Check if the firewall rules, forwarding rules, and the gke-oidc-envoy resources are properly referenced in Terraform.

2. Manually Remove Stale Resources Before Destroying

- If Terraform isn't handling deletions properly, try manually removing leftover resources before destroying the VPC.

3. Check for Terraform State Issues

- If Terraform is not correctly recognizing the resources, refresh the state. If any resources are missing, import them and then try to destroy the infrastructure again.

4. Ensure Proper Terraform Provider Version

- If the issue started after a provider upgrade, check which versions worked before and compare the changes. If necessary, downgrade the provider version in versions.tf and then reinitialize Terraform.

5. Retry Terraform Destroy

- After fixing dependencies and cleaning up stale resources, try running again.

If you need further assistance, you can reach out to Google Cloud Support at any time.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[yossig-runai]

Hi, thanks for the response, alao the load balancer is created as internal instead of external, what are the needs for creating it as external, because the cluster is having a public ip

[nmagcalengjr]

To create the gke-oidc-envoy load balancer as external instead of internal, you’ll need to modify how GKE deploys the OIDC authentication components. By default, Google Cloud creates an internal load balancer for OIDC, but you can override this by modifying the service settings or manually configuring an external LoadBalancer.

1. Verify the current service type, as gke-oidc-envoy is deployed as an internal LoadBalancer by default.

kubectl get service gke-oidc-envoy -n kube-system -o yaml

2. To use an external load balancer for gke-oidc-envoy, update the service by removing the internal annotation and specifying an external load balancer.
3. Enable internet traffic by checking if GKE has automatically created a firewall rule blocking external access and updating it as needed.

• List Firewall Rules

gcloud compute firewall-rules list --filter="name:gke-oidc-*"

• Update the Firewall Rule to Allow Public Access

Modify the firewall rule to allow public access to the load balancer:

gcloud compute firewall-rules update FIREWALL_RULE_NAME --allow tcp:443 --source-ranges=0.0.0.0/0

Or, if needed, create a new rule:

gcloud compute firewall-rules create allow-external-oidc \

    --network=YOUR_NETWORK_NAME \

    --allow tcp:443 \

    --source-ranges=0.0.0.0/0 \

    --target-tags=gke-oidc-envoy

4. Verify the External Load Balancer to ensure the service has an external IP after applying the changes. The assigned IP should be external rather than internal.

kubectl get svc gke-oidc-envoy -n kube-system

5. Update the OIDC configuration to use the new external endpoint. If your applications were previously relying on the internal OIDC service, update their settings to use the external IP and adjust the OIDC provider configuration if needed.

export OIDC_EXTERNAL_IP=$(kubectl get svc gke-oidc-envoy -n kube-system -o jsonpath='{.status.loadBalancer.ingress[0].ip}')

echo "External OIDC Load Balancer IP: $OIDC_EXTERNAL_IP"

To expose gke-oidc-envoy externally, remove the internal load balancer annotation, set the service type to LoadBalancer, and ensure firewall rules allow public access.

If you need further assistance, you can reach out to Google Cloud Support at any time.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.