Connection refused for metadata server when Dataplane V2 enabled in GKE

[ganeshk-searce]

Hi

I am getting connection refused error metadata server when dataplane v2 is enabled for GKE cluster. More 

Setup:

• A regional GKE standard cluster with public endpoint and private nodes enabled
• VPC is associated with NAT Gateway to allow egress from GKE Nodes
• Dataplane V2 option is enabled
• Workload Identity is configured with GCP SA with Cloud KMS Admin role
• A GoLang app which can create and read KMS keys

Problem

• When there is no NetworkPolicy used for GoLang app associated pod, I am seeing the following error.
  
  

  rpc error: code = Unauthenticated desc = transport: per-RPC creds failed due to error: Get \"http://169.254.169.254/computeMetadata/v1/instance/service-accounts/default/token?scopes=https%3A%2F...": dial tcp 169.254.169.254:80: connect: connection refused

• As per the third statement in this link, I have attached a NetworkPolicy to allow egress for IP Range 169.254.169.254/32 on port 80.  (GKE Cluster version: 1.21.14-gke.2700). Now, I am seeing a different error as below,
  

  
  google: could not find default credentials. See https://developers.google.com/accounts/docs/application-default-credentials for more information.
  

Please suggest me to address this in right way.

Thanks in advance

[dasalemi]

I have a few questions: Did you follow the link? Did you create the Application defaults credentials? Did the system show an error message or warning when you try to create ADC? Please elaborate a little more.

Here is more information about providing credentials for Application Default Credentials.