GKE node upgrade stuck "Error 400: Cluster is running incompatible operation operation"

[dilshanonline]

Hello,

I have a GKE cluster which was trying to upgrade it's nodes. But seems the operation got stuck and I can see the below error for the last 2 days.

When I try to run any command using the CLI it will give me the below error,

ERROR: (gcloud.container.clusters.update) FAILED_PRECONDITION: Cluster is running incompatible operation operation-1700226020173-210a422c-f4b9-41f0-9e21-f9bee66c5535.
- '@type': type.googleapis.com/google.rpc.RequestInfo
  requestId: '0x290efa3770b767d8'

gcloud container operations list command giving an output telling the operation still aborting.

023-11-17T12:53:32.0282925Z    2023-11-17T12:53:35.327130441Z
operation-1700226020173-210a422c-f4b9-41f0-9e21-f9bee66c5535  UPGRADE_NODES       europe-west4  example-cluster-node-pool-1                                                                                                                                                                                                                                                                                                                                                                                                                         ABORTING  2023-11-17T13:00:20.173869352Z

Since this, I cannot edit the cluster in the console as well. If i try to redeploy using terraform it will timeout telling the above : googleapi: Error 400: Cluster is running incompatible operation operation error. 

Appreciate your feedback to resolve this.

[lawrencenelson]

Hi @dilshanonline,

Welcome to the Google Cloud Community!

Since Google Kubernetes Engine (GKE) is a managed service, and Google is the one responsible for upgrading the control plane and node version, it would be better to contact Google Cloud Support directly so they can check the backend.

I hope this helps. Thank you. 😃