Why are new nodes using an old version.

pdamer · 04-24-2024 08:04 AM

I am trying to upgrade my autopilot GKE cluster above 1.29.2-gke.1060000 so that I can use Bursting workloads. I've upgraded the cluster itself to 1.29.3-gke.1093000 but the nodes are still at 1.27.8-gke.1067004. I tried cordoning off a node so it would get replaced but the new node was still on the old 1.27.8-gke.1067004 version.

Is there a way to force the nodes to be upgraded past 1.29.2 in autopilot?

aas1

I'm seeing the same thing 🤔

Running GKE Autopilot with STABLE release channel.
Control plane is upgraded to `v1.30.3-gke.1969001` but the nodes are running `v1.29.7-gke.1104000`

lukes

I am affected as well, GKE Autopilot:

gcloud container clusters list
NAME                            LOCATION      MASTER_VERSION      MASTER_IP     MACHINE_TYPE  NODE_VERSION        NUM_NODES  STATUS
production-k8s-cluster-ce560b4  europe-west4  1.30.5-gke.1014001  x.x.x.x       e2-medium     1.29.1-gke.1589018  64         RUNNING

clintonb-vori

Five months later, and this remains an issue. My control plane auto-updated to `v1.30.10-gke.1022000`, but pod disruption budgets prevented some nodes from upgrading. However, even after removing the PDBs, the overwhelming majority of my nodes are on `v1.30.9-gke.1046000`. In fact, while typing this, I have now lost one of the newer nodes!

clintonb-vori

Ignore my previous comment as I misread the valid version number. I opened a support request, and GCP identified an orphaned node pool on an older k8s version. It took a few days, but they eventually removed the orphaned node pool and I was able to enable bursting after re-upgrading my cluster.