Hello everyone!
I'm having issues with implementing GKE Workload Identity configuration, FYI these are a 3 years old cluster that has been upgraded to Kubernetes version 1.30.
The error logs from gke-metadata-server pods as follows:
```
I1116 17:02:40.271432 1949 federated_token.go:91] [conn-id:619a304a9b4f923f rpc-id:0ff2d363617493ff background-id:a5440cd568acd98c remote-addr:10.130.0.55:60908 pod:vault/vault-2] Fetching federated token for pod vault/vault-2
I1116 17:02:40.384629 1967 serviceaccounts.go:164] [conn-id:619a304a9b4f923f rpc-id:0ff2d363617493ff remote-addr:10.130.0.55:60908 pod:vault/vault-2] Failed to fetch token for pod vault/vault-2: generic::invalid_argument: loading: getFederatedToken("vault-access@project-name.iam.gserviceaccount.com", "", "", vault/vault-2): loading: while exchanging k8s token for federated token: failed to generate federated token using STS: googleapi: got HTTP response code 400 with body: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}
E1116 17:02:40.384720 1967 metadata.go:225] [conn-id:619a304a9b4f923f rpc-id:0ff2d363617493ff remote-addr:10.130.0.55:60908 pod:vault/vault-2] "/computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform%2Chttps%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloudkms" HTTP/400: generic::invalid_argument: loading: getFederatedToken("vault-access@project-name.iam.gserviceaccount.com", "", "", vault/vault-2): loading: while exchanging k8s token for federated token: failed to generate federated token using STS: googleapi: got HTTP response code 400 with body: {"error":"invalid_target","error_description":"The target service indicated by the \"audience\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist."}, Reason: "STS", UserMessage: "400 Bad Request: {\"error\":\"invalid_target\",\"error_description\":\"The target service indicated by the \\\"audience\\\" parameters is invalid. This might either be because the pool or provider is disabled or deleted or because it doesn't exist.\"}", started at 2024-11-16 17:02:40.270276091 +0000 UTC m=+3460.622698067
```
These are the step that I follow Authenticate to Google Cloud APIs from GKE workloads and I have implemented this before in a different cluster and it works perfectly.
Things that I tried:
- Create a new Node Pool with GKE Metadata Server enabled and adding taint to isolate the pod from being deployed in the old Node Pool.
- Adding Network policies to enable egress to the gke-metadata-server.
- Create a new GKE Cluster with Workload Identity enabled and testing it with these steps above and it works.
- Comparing the gke-metadata-server daemon set configuration (Similar Config)
- Comparing the gke cluster configuration (Similar Config)*
- Comparing the gke node pool configuration (Similar Config)*
Notes:
* The only difference is the 3 year old cluster are having issues with workload identity, but creating a new cluster are working just fine with these features.
Thank you.
Good day.
** UPDATES 26 FEB 2025 **
Found some differences in the `Command` flag of gke-metadata-server DaemonSet configuration between new cluster that is working
```
--identity-provider=https://container.googleapis.com/v1/projects/projec-name/locations/asia-southeast2-a/clusters/new-cluster
```
LinkedIn
Twitter
