Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.

GKE Workload Identity return 400 when accessing Google Cloud resources

Hello everyone!

I'm having issues with implementing GKE Workload Identity configuration, FYI these are a 3 years old cluster that has been upgraded to Kubernetes version 1.30.

The error logs from gke-metadata-server pods as follows:

```

I1116 17:02:40.271432 1949 federated_token.go:91] [conn-id:619a304a9b4f923f rpc-id:0ff2d363617493ff background-id:a5440cd568acd98c remote-addr:10.130.0.55:60908 pod:vault/vault-2] Fetching federated token for pod vault/vault-2
I1116 17:02:40.384629 1967 serviceaccounts.go:164] [conn-id:619a304a9b4f923f rpc-id:0ff2d363617493ff remote-addr:10.130.0.55:60908 pod:vault/vault-2] Failed to fetch token for pod vault/vault-2: generic::invalid_argument: loading: getFederatedToken("vault-access@project-name.iam.gserviceaccount.com", "", "", vault/vault-2): loading: while exchanging k8s token for federated token: failed to generate federated token using STS: googleapi: got HTTP response code 400 with body: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}
E1116 17:02:40.384720 1967 metadata.go:225] [conn-id:619a304a9b4f923f rpc-id:0ff2d363617493ff remote-addr:10.130.0.55:60908 pod:vault/vault-2] "/computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloudkms" HTTP/400: generic::invalid_argument: loading: getFederatedToken("vault-access@project-name.iam.gserviceaccount.com", "", "", vault/vault-2): loading: while exchanging k8s token for federated token: failed to generate federated token using STS: googleapi: got HTTP response code 400 with body: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}, Reason: "STS", UserMessage: "400 Bad Request: {\"error\":\"invalid_target\",\"error_description\":\"The target service indicated by the \\\"audience\\\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist.\"}", started at 2024-11-16 17:02:40.270276091 +0000 UTC m=+3460.622698067

```

These are the step that I follow Authenticate to Google Cloud APIs from GKE workloads and I have implemented this before in a different cluster and it works perfectly.

Things that I tried:

  1. Create a new Node Pool with GKE Metadata Server enabled and adding taint to isolate the pod from being deployed in the old Node Pool.
  2. Adding Network policies to enable egress to the gke-metadata-server.
  3. Create a new GKE Cluster with Workload Identity enabled and testing it with these steps above and it works.
  4. Comparing the gke-metadata-server daemon set configuration (Similar Config)
  5. Comparing the gke cluster configuration (Similar Config)*
  6. Comparing the gke node pool configuration (Similar Config)*

Notes:

* The only difference is the 3 year old cluster are having issues with workload identity, but creating a new cluster are working just fine with these features.

Thank you.

Good day.

 

** UPDATES 26 FEB 2025 **
Found some differences in the `Command` flag of gke-metadata-server DaemonSet configuration between new cluster that is working

```

--identity-provider=https://container.googleapis.com/v1/projects/projec-name/locations/asia-southeast2-a/clusters/new-cluster

 

```

0 2 637
2 REPLIES 2

Just to double-check, does the 3 year old cluster have Workload Identity enabled?

Yes, the feature is enabled and each node already have the `iam.gke.io/gke-metadata-server-enabled=true` annotations

Top Labels in this Space