GKE pod metrics not collected, timeout when accessing kubelet

[Justas]

Control plane version: `1.30.8-gke.1162001`

Node pool version: `1.28.2-gke.1157000`

I recently upgraded the node pool Kubernetes version and after the upgrade there are no metrics displayed in the dashboard. I see these logs in the metrics-server pod:

```

"Failed to scrape node, timeout to access kubelet" err="Get \"https://10.0.0.33:10250/metrics/resource\": context deadline exceeded" node="<name>" timeout="10s"

```

In the features section cloud monitoring was enabled for pods and deployments, but that did not seem to help.

[debolek]

Since GKE integrates with Cloud Logging, check logs in Google Cloud Console and paste it here  :- gcloud logging read "resource.type=k8s_container ANDresource.labels.cluster_name=YOUR_CLUSTER_NAME" --limit 50

[Justas]

Maybe there are more specific services I should be looking at? That command mainly returns logs from the applications that are running in the cluster.

I noticed this issue in another cluster as well. It started on February 15th. But enabling cloud monitoring for pods and deployments seemed to help resolve it.

Another thing I noticed is that it only seems to happen with spot node pools. I upgraded a standard node pool to version 1.28.2-gke.1157000 in the same cluster yesterday and metrics for pods running in that node pool are still shown.