Cannot Access Metadata from Autopilot Cluster

[JLloyd]

Hello all,

I am working on changing our server from a Standard Cluster to an Autopilot Cluster, but am running into an issue.

We store the IP of our server in the Compute Engine => Metadata section where we can have key/value pairs for sitewide data.

Google documents say to access this with the following command:

curl -s http://metadata.google.internal/computeMetadata/v1/project/attributes/server_ip -H "Metadata-Flavor: Google"

This has worked great for years, and we use it in several scripts in our workloads to pull down the stored IP.

However, once the workload is running on an Autopilot cluster, this command now returns a 404 not found.
I'm not finding anything online about this issue anywhere, and don't know what to do to proceed.  How do I get workloads on the Autopilot cluster to be able to access this information?
Any help would be greatly appreciated.

Thanks

[lawrencenelson]

Hi @JLloyd,

Welcome to the Google Cloud Community!

Can you share the exact error that you're getting? Can you also try spinning up a new standard cluster and check if you're getting the same results?

This might be resolved by setting up Workload Identity. You may check out this Medium article.

Best,
Lawrence

[JLloyd]

Hello,
Thank you for the reply.  Much appreciated.
Yes, if I revert back to a standard cluster, it works immediately with no issues.  There's no setup or fuss, it just works.
But going back to an autopilot cluster and I get a 404 for the metadata url.

Thanks for the link on the article, I'll read through it.

I edited my workload to have a verbose output on the curl command, and this is what it gets back:

curl -v http://metadata.google.internal/computeMetadata/v1/project/attributes/server_ip -H "Metadata-Flavor: Google"

% Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                   Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*
  Trying 169.254.169.254...
  * TCP_NODELAY set
  * Connected to metadata.google.internal (169.254.169.254) port 80 (#0)  
> GET /computeMetadata/v1/project/attributes/server_ip HTTP/1.1
> Host: metadata.google.internal
> User-Agent: curl/7.52.1
> Accept: */*
> Metadata-Flavor: Google
>
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Wed, 08 Nov 2023 21:54:49 GMT
< Content-Length: 10
<
{ [10 bytes data]
* Curl_http_done: called premature == 0
100    10  100    10    0     0   1024      0 --:--:-- --:--:-- --:--:--  1111
* Connection #0 to host metadata.google.internal left intact
Not Found

[lawrencenelson]

Hi @JLloyd,

---

@JLloyd wrote:

We store the IP of our server in the Compute Engine => Metadata section where we can have key/value pairs for sitewide data.

---

Just to clarify, did you create these VMs manually in the Compute Engine page? Or are you referring to VMs that were created when you created the cluster?

[JLloyd]

The server is on a VM that we created using the Compute Engine.  The IP for that server is stored in Metadata.

[garisingh]

Autopilot clusters use Workload Identity (you can't disable it) and when Workload Identity is enabled, you have access to the GKE Metadata server which only makes a subset of the metadata from the GCE Metadata server available.

[JLloyd]

Thanks for the response.  However, I'm not sure what to gather from that.  That resource is where I originally got the url from that I put in my post.  According to that document, the Workload Identity should have access to the "Project metadata".  

https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#project_metadata

So why does this url work with a Standard cluster but not with an Autopilot cluster that has the Workload Identity turned on by default?

curl -s http://metadata.google.internal/computeMetadata/v1/project/attributes/server_ip -H "Metadata-Flavor: Google"

[garisingh]

if you look at 

https://cloud.google.com/kubernetes-engine/docs/concepts/workload-identity#project_metadata

you will see that only two metadata entries - project-id and numeric-project-id - are accessible through the GKE metadata server used by Workload Identity.  

[garisingh]

The GKE Metadata Server only has a subset of the GCE metadata server entries.

[JLloyd]

So, how is it intended to be able to access Compute Engine => Metadata entries from an autopilot cluster with Workload Identity enabled?  

This url, which is linked from the Metadata page, does not say anything about it that I can find.

https://cloud.google.com/compute/docs/metadata/querying-metadata

[JLloyd]

I'm still no further on this topic.  Hopefully someone still has some ideas.