GKE: metric server crashlooping (crosspost from r/googlecloud)

[LucaGervasi]

I have several (<10) gke clusters, all but one are all in the same condition and I can't figure out what and why is it happening. I hope to find someone that managed to solve the same issue 🙂

Some time ago, i noticed that our HPA stopped working, having no way to read metrics from pods. Long story short, our pod named "metrics-server-v0.5.2-*" crashloops outputting a stack trace like this one: https://nopaste.net/aA5rwAeBWI and this one: https://nopaste.net/3uQsD6EDfA .

I tried to restart the deployment, without any success. What I don't understand is why one of the clusters (the oldest to be created) is working...

From r/googlecloud (reddit) it seems aknowledged that is not specifically tied to our workloads.

All the clusters are updated to the same version: v1.24.12-gke.500 and have been created by terraform resources.

Do any of you have any pointer?

Thanks.

[garisingh]

Any chance your GKE API server certificate(s) have expired?

[LucaGervasi]

Negative.

Btw, the master is managed by Google, so I think that is not even possible.

[SachinKaladin]

We are also observing an issue with 1.25.13-gke.200. We have two clusters with the same k8s version and the component version, but issue happens in only one of them. Since it is gke managed,  any idea how we can start looking to help us to address the issue

I0928 14:50:45.579259 1 shared_informer.go:247] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x4662bf]

goroutine 998 [running]:
runtime.throw({0x1b7bd32?, 0x1?})
/usr/local/go/src/runtime/panic.go:992 +0x71 fp=0xc000824a70 sp=0xc000824a40 pc=0x4360d1
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:825 +0x2ec fp=0xc000824ac0 sp=0xc000824a70 pc=0x44c30c
aeshashbody()
/usr/local/go/src/runtime/asm_amd64.s:1343 +0x39f fp=0xc000824ac8 sp=0xc000824ac0 pc=0x4662bf
runtime.mapiternext(0xc0000c5e40)
/usr/local/go/src/runtime/map.go:934 +0x2cb fp=0xc000824b38 sp=0xc000824ac8 pc=0x40fc4b
runtime.mapiterinit(0x8604a2?, 0xc000971640?, 0xc000549940?)
/usr/local/go/src/runtime/map.go:861 +0x228 fp=0xc000824b58 sp=0xc000824b38 pc=0x40f928
reflect.mapiterinit(0x874dd6?, 0x1915840?, 0xc0000c5e40?)
/usr/local/go/src/runtime/map.go:1373 +0x19 fp=0xc000824b80 sp=0xc000824b58 pc=0x462ab9
github.com/modern-go/reflect2.(*UnsafeMapType).UnsafeIterate(...)
/go/pkg/mod/github.com/modern-go/reflect2@v1.0.1/unsafe_map.go:112
github.com/json-iterator/go.(*sortKeysMapEncoder).IsEmpty(0x11?, 0x174980?)
/go/pkg/mod/github.com/json-iterator/go@v1.1.10/reflect_map.go:333 +0x28 fp=0xc000824bc0 sp=0xc000824b80 pc=0x868288
github.com/json-iterator/go.(*placeholderEncoder).IsEmpty(0x203000?, 0xc000e80020?)