GKE can't perform maintenance because the pod disruption budget allows for 0 pod evictions. Update t

[ankit_rawat28]

I have created GCP Composer-2 (Airflow) from UI and it has created GKE Autopilot by itself to manage dags tasks.
Now after sometime using it i have started noticing that on GKE Page i am receiving a warning as below.

Fix unpermissive pod disruption budget.
GKE can't perform maintenance because the pod disruption budget allows for 0 pod evictions. Update the pod disruption budget.

I have also attached a official google doc where it is mentioned that Possible solutions: Ignore these warnings until the issue is fixed.

Is there anyway to resolve this or it is expected behaviour?

 Insight

A pod disruption budget exists that allows for 0 pod evictions. A pod disruption budget must allow for at least 1 pod eviction so that GKE can perform maintenance. Learn more 

Cluster name: europe-west2-airfllow-8-gke

Location: europe-west2

Unpermissive pod disruption budgets

NameExpected podsSpecValueRecommended Value

airflow-database-init-job-pdb	0	maxUnavailable	0	1
airflow-redis-pdb	1	maxUnavailable	0	1
airflow-webserver-pdb	1	minAvailable	1	0
airflow-worker-pdb	1	maxUnavailable	0	1
composer-agent-pdb	0	maxUnavailable	0	1
composer-snapshots-pdb	0	maxUnavailable	0	1

[pgstorm148]

You've encountered a common issue with GKE Autopilot clusters automatically created by Cloud Composer 2: overly restrictive Pod Disruption Budgets (PDBs). Let's break down the problem and what you can do.

Understanding the Issue

• Pod Disruption Budgets (PDBs):
  • PDBs are Kubernetes objects that limit the number of pods that can be voluntarily disrupted at any one time.
  • They're designed to ensure application availability during planned disruptions, such as node maintenance or updates.
• Composer 2 and GKE Autopilot:
  • When you create a Composer 2 environment, it automatically provisions a GKE Autopilot cluster to run Airflow tasks.
  • The default PDBs created by Composer 2 are often overly restrictive, especially in Autopilot mode, which can hinder GKE maintenance.
• GKE Maintenance:
  • GKE needs to perform periodic maintenance, including node updates and security patches.
  • If PDBs are too restrictive, GKE cannot evict pods to perform maintenance, leading to the warning you're seeing.
• The Problem:
  • The main problem is that the PDBs that are created have a value of 0 for maxUnavailable, which prevents GKE from evicting any pods.

Why Google Suggests Ignoring (Temporarily)

• Google's recommendation to "ignore these warnings until the issue is fixed" likely stems from the fact that:
  • They are aware of this behavior in Composer 2's Autopilot setups.
  • They are working on a more permanent solution to automatically adjust PDBs or improve the default configuration.
  • That changing the PDBs manually, could cause unexpected issues with the composer environment.

How to Resolve (With Caution)

While Google's advice is to wait, you can manually adjust the PDBs. However, proceed with caution and understand the potential risks.

1. Identify the PDBs:

   • Use kubectl get pdb -n <your-composer-namespace> to list the PDBs in your Composer environment's namespace.
   • The namespace will be something like composer-&lt;your-environment-name>.

2. Edit the PDBs:

   • Use kubectl edit pdb -n <your-composer-namespace> <pdb-name> to edit each PDB.
   • Change the maxUnavailable or minAvailable values to allow for at least one pod eviction.
     • For example, change maxUnavailable: 0 to maxUnavailable: 1.
     • Or, change minAvailable: 1 to minAvailable: 0.
   • It is generally safer to modify the maxUnavailable value.
   • It is also possible to patch the pdb, by using kubectl patch.
   • Example:
     • kubectl patch pdb airflow-redis-pdb -n composer-<your-environment-name> -p '{"spec":{"maxUnavailable": 1}}'

3. Monitor GKE Maintenance:

   • After adjusting the PDBs, monitor your GKE cluster to ensure that maintenance operations can proceed without issues.
   • Keep an eye on the GKE console for any new warnings or errors.

Important Considerations

• Composer Updates:
  • Be aware that future Composer environment updates might overwrite your manual PDB changes.
  • You might need to reapply the changes after updates.
• Application Availability:
  • Adjusting PDBs can affect the availability of your Airflow components.
  • Ensure that your applications can tolerate the potential disruptions during maintenance.

[ankit_rawat28]

Thanks @pgstorm148  for the response. 
I was thinking the same. doubt is clear now.