GKE standard cluster - pods creation failure - more than one default storage class was found

[Kanipalanisamy]

Hi,
I had a GKE cluster with a stateful set containing 3 pods with a persistent volume claim each. The pods were torn down by GKE on Jul 27th and they were not created successfully. I was able to notice the following error in the cluster audit log - 

"persistentvolumeclaims ""data-<pod name>"" is forbidden: Internal error occurred: 2 default StorageClasses were found" 

I did not change the storage class annotation, `storageclass.kubernetes.io/is-default-class` for any of the existing storage classes in the cluster. I don't understand how there were 2 default storage classes suddenly. It would be very helpful to understand how it happened.

To facilitate the pod creation, I changed the `storageclass.kubernetes.io/is-default-class` annotation from `true` to `false` for one of the storage classes. I picked the one that wasn't supposed to be marked `true` by comparing it with another healthy cluster of mine. Although, the PVCs and the pods were successfully created, the data from my old persistent volumes were lost. I had to redeploy my stateful set and reconfigure.

[dionv]

Hi @Kanipalanisamy 

Per my investigation, it seems that this is a known error with 2 default Storage classes found. In this link, there was concern raised way back in 2017. And in this link, there was a discussion on how to fix it, as per thockin choosing either one of the 2 default storage class is fine to resolve the concern.

[Kanipalanisamy]

Thanks @dionv . It is good to know about the resolution.
May I ask if there was any rollout on the GKE side that could have ended up causing 2 default storage classes in my cluster? My cluster is on stable release channel running 1.25.10-gke.1200 right now but I don't know the version it was on before the GKE maintenance on Jul 27th.