GKE cluster type: standard
GKE cluster version: 1.32.3-gke.1057001
GKE cluster region: us-central1
issue:
kube-dns sidecar container is using a non-existing image `us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20250317_2300_RC0@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19`, which causing all DNS queried failed in the cluster
Pod events as follows:
```
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 15m default-scheduler no nodes available to schedule pods
Warning FailedScheduling 14m (x7 over 15m) default-scheduler no nodes available to schedule pods
Warning FailedScheduling 12m (x10 over 14m) default-scheduler no nodes available to schedule pods
Normal Scheduled 12m default-scheduler Successfully assigned kube-system/kube-dns-6dbf6d787c-qvfbc to gke-gcp-usc1-aq5bs-s-default-node-poo-739b86e9-wb95
Normal NotTriggerScaleUp 14m cluster-autoscaler pod didn't trigger scale-up:
Warning FailedCreatePodSandBox 12m kubelet Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox "fbcb892138f0b2fe6551031f07c4a1f586df892acae258bec51247a49a814d43": plugin type="calico" failed (add): error getting ClusterInformation: resource does not exist: ClusterInformation(default) with error: clusterinformations.crd.projectcalico.org "default" not found
Normal Pulling 12m kubelet Pulling image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-kube-dns:1.25.0-gke.4@sha256:06837d58557deb555dfc26937cc45a32a0726a2e94f5b7c23a168273e544a077"
Normal Pulled 12m kubelet Successfully pulled image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-kube-dns:1.25.0-gke.4@sha256:06837d58557deb555dfc26937cc45a32a0726a2e94f5b7c23a168273e544a077" in 2.772s (12.757s including waiting). Image size: 33004697 bytes.
Normal Pulling 12m kubelet Pulling image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-dnsmasq-nanny:1.25.0-gke.4@sha256:5c86940a8b0578cd97f113da10850956f55b66d88a8899dfc2c60ad3fdca3ea6"
Normal Started 12m kubelet Started container kubedns
Normal Created 12m kubelet Created container: kubedns
Normal Pulled 11m kubelet Successfully pulled image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-dnsmasq-nanny:1.25.0-gke.4@sha256:5c86940a8b0578cd97f113da10850956f55b66d88a8899dfc2c60ad3fdca3ea6" in 6.233s (17.542s including waiting). Image size: 37099930 bytes.
Normal Created 11m kubelet Created container: dnsmasq
Normal Started 11m kubelet Started container dnsmasq
Normal Pulling 11m kubelet Pulling image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-sidecar:1.25.0-gke.4@sha256:e6adc0cf878edaa70778582684a97ac744e2e88b69de6f3d4b536e3dd68fe1fd"
Normal Pulled 11m kubelet Successfully pulled image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/k8s-dns-sidecar:1.25.0-gke.4@sha256:e6adc0cf878edaa70778582684a97ac744e2e88b69de6f3d4b536e3dd68fe1fd" in 2.584s (2.584s including waiting). Image size: 29328126 bytes.
Normal Created 11m kubelet Created container: sidecar
Normal Started 11m kubelet Started container sidecar
Normal Pulling 10m (x4 over 11m) kubelet Pulling image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20250317_2300_RC0@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19"
Warning Failed 8m48s (x5 over 11m) kubelet Failed to pull image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20250317_2300_RC0@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19": rpc error: code = NotFound desc = failed to pull and unpack image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19": failed to resolve reference "us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19": us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19: not found
Warning Failed 8m48s (x5 over 11m) kubelet Error: ErrImagePull
Warning Failed 2m26s (x42 over 11m) kubelet Error: ImagePullBackOff
Normal BackOff 2m14s (x43 over 11m) kubelet Back-off pulling image "us-central1-artifactregistry.gcr.io/gke-release/gke-release/gke-metrics-collector:20250317_2300_RC0@sha256:a34349ad63db36e11c2055578c3d2ef08bee75be14a2ee3cb244d9ddf74f0c19"
```