Google Anthos on VMware (1.16) - Deleting a particular node

[SrinathOR]

Dear Community members,

We are using Google Anthos for the past couple of years now. Currently, each clusters have two node pools, 1. npool-large  2. npool-xlarge.

We aim to increase the number of nodes in the X-large node pool and eventually decrease it in npool-large node pool. However, we need to handle this in a much controlled manner.

We cordon/drained particular nodes, however when I decrease the number of the replicas in node pool npool-large from 3 to 2 in cluster configuration and run gkectl update cluster --kubeconfig /path --config <cluster-config.yaml> , it is not deleting the node which is cordoned or drained, it is deleting the healthy nodes from node pool npool-large.

Is there any way that we can choose which node to be deleted in google Anthos using gkectl update command ?

Thanks

Srinath

[shayan]

Hi @SrinathOR,

1. Node Restarts During Update:

It's normal for all nodes to restart when you update node pools.

2. Understanding Node Pool Management:

According to my understanding:

A. You must have at least one more node (IP) than the requirement.

https://cloud.google.com/kubernetes-engine/distributed-cloud/vmware/docs/how-to/managing-node-pools#...

B. Follow these steps for node pool updates:

IMPORTANT: You must use labels for node pools. Like runtime and data nodes.

1. Cordon the Node:

kubectl cordon <node-name>

2. Drain the Node:

kubectl drain <node-name> --ignore-daemonsets --delete-local-data

3. Delete the Node:

kubectl delete node <node-name>

4. Adjust Node Pool Size:

Adjust the size of the node pool from where you want to remove the node.

5. Update Cluster Configuration:

Run the following command:

gkectl update cluster --kubeconfig /path/to/kubeconfig --config <cluster-config.yaml>

6. Increase Node Pool Replicas:

Adjust the size of the node pool for which you want to increase replicas.

7. Update Cluster Again:
Run the command again:

gkectl update cluster --kubeconfig /path/to/kubeconfig --config <cluster-config.yaml>

Hope this works for you.

[SrinathOR]

Hi Shayan

Thanks, I have already tried

1. cordon/drain --> resizing the cluster configuration --> gkectl update.. however, observed that the node which is cordoned or drained is not getting deleted, the update command picks a random node and deletes it 

2. Cordon/drain--> manually deleting a node : creates a new node immediately. Even if I resize cluster configuration and run gkectl command its of no use now, as It might delete new node or randomly picks an old node.

My idea is to safely drain the node to move all the running pods from Large to  x-large node pool, and when i shrink the replica size of large node pool then running gkectl update command  it should remove the node which is drained and this is expected behaviour