Reduce node pool size - control which nodes get removed

[ravens]

We are running a GKE cluster which has 5 nodes. Two of the nodes do not host a service of ours.
The node pool is currently configured to have 5 nodes.
Now we want to reduce the size of the node pool to 3.
Can I somehow control which of the nodes are being deleted? For example, can I cordon two of the nodes which are then being deleted when I reduce the node pool size to 3?

[rayjohnn]

Good day!

If you have a specific node that you wan to keep, here are the steps I recommend you take.

First is cordon the nodes that you need then drain the rest to get to the target node pool size which is 3.

Command to cordon:

kubectl cordon <NODE_NAME_TO_KEEP>

Command to drain:

kubectl drain <NODE_NAME_TO_DELETE> --ignore-daemonsets --delete-emptydir-data

Once completed you may now resize the node pool.

gcloud container clusters resize <CLUSTER_NAME> --node-pool <NODE_POOL_NAME> --num-nodes=3

Just a friendly reminder to backup all your data before undergoing this process. Attached are documentations for future reference.[1][2]

[1] https://kubernetes.io/docs/tasks/administer-cluster/safely-drain-node/
[2] https://cloud.google.com/kubernetes-engine/docs/how-to/resizing-a-cluster

[yuanz36]

Thanks for this. 

Wondering what is the purpose of 

kubectl cordon <NODE_NAME_TO_KEEP>

And how it would affect the GCP/GKE's scale-in strategy?

[mahmoudrabie]

Hi @ravens 

you can influence which nodes are removed by cordoning them and then draining them, although this process requires a bit of manual intervention. Here's how you can approach this:

(1) Cordon the nodes

Start by cordoning the nodes you want to be removed. This prevents new pods from being scheduled on these nodes but does not affect existing pods.

kubectl cordon <node-name>

(2) Drain the nodes

After cordoning, you'll need to drain the nodes, which safely evicts all pods from the node in preparation for removal. This step is crucial for ensuring that the services hosted on these nodes are not abruptly terminated.

kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data

 (3) Reduce the Node Pool Size

Once the desired nodes have been cordoned and drained, you can proceed to reduce the node pool size. This can be done through the GCP Console or using the gcloud command-line tool.

gcloud container clusters resize <cluster-name> --node-pool <node-pool-name> --num-nodes 3