Transient invalid response from metadata service: Metadata-Flavor header

[waynep]

In our GKE workload, we will sporadically receive an error response (see below) from the metadata service. This is coming from the gcp-metadata package (source code).

Our GKE is located in us-central-1 and nodes were previously in a,b,c (eg. us-central-1a, us-central-1b, etc).

Last week, we saw the node in us-central-1c exhibit this behavior on some workloads. We balanced out of 1c into a, b, f. 

The issue re-appeared in 1f a week later.

Given this is not consistent and from the gcp-metadata dependency... looking for solutions or should this be raised to the GCP team?

Error: Invalid response from metadata service: incorrect Metadata-Flavor header.
at metadataAccessor (/opt/app/node_modules/@google-cloud/storage/node_modules/gcp-metadata/build/src/index.js:120:19)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at async GoogleAuth.getUniverseDomainFromMetadataServer (/opt/app/node_modules/@google-cloud/storage/node_modules/google-auth-library/build/src/auth/googleauth.js:169:30) 

[yotsumi · Accepted solution]

ℹ️A workaround is available in this github issue

[waynep]

thanks! incredibly didn't find that GH issue when I searched that repo.

(didn't get notification so missed your response earlier)

[reinc]

Hi @waynep, 

Welcome to Google Cloud Community! 

I agree with @yotsumi workaround. To make it summarize on the thread @DmitrijBuckovsky mentioned that updating google-auth-library to latest (9.15.1) and version 9.12.0 works on him by adding "google-auth-library": "^9.15.1" on package.json.

Modify the dependencies section of your package.json file to include the following:

"dependencies": {
  "google-auth-library": "^9.15.1",

}

Note: You can also use ^9.12.0 or any later version as needed.

Run the following command to update your package-lock.json file and install the updated google-auth-library package:

npm i

This command will install the specified version of google-auth-library and update your package-lock.json file to ensure consistent dependency resolution across environments.

For your reference, you may check here the General Google Cloud Documentation on Authentication and GKE:

• Authentication methods at Google 
• GKE Workload Identity
• Troubleshooting GKE Networking

If you need further assistance and any questions, please reach out to our Google Cloud Support team.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.

[yotsumi]

Little tips: you're not required to include the dependency, you can just force the resolution of the right with version when the dependency is required as a transitive dependency.

package.json

"resolutions": {
  "google-auth-library": "^9.15.1"
}