This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Pod eviction error in Cloud Composer

Posted on 11-01-2023 02:42 AM
kabilanravi
Bronze 3
Bronze 3
Reply posted on --/--/---- --:-- AM

In our cloud composer v2, occasionally we are getting pod eviction error from its kubernetes cluster.

Error: The task might not have been executed or worker executing it might have finished abnormally (eg, was evicted)

How can we solve this issue? Any help is much appreciated.

1 2 1,914
Topic Labels
2 REPLIES 2

Posted on --/--/---- --:-- AM
Poala_Tenorio
Staff
Reply posted on --/--/---- --:-- AM

The error message you provided, "Error: The task might not have been executed or the worker executing it might have finished abnormally (e.g., was evicted)," typically indicates an issue with the execution of a task within your Apache Airflow DAG in a Cloud Composer environment. This could be due to various reasons, with pod eviction being one of them. Pod eviction is when a Kubernetes pod (the smallest deployable unit in a Kubernetes cluster) is forcibly terminated and removed from a node. This can happen for several reasons, such as resource constraints, node failures, or other issues.

To resolve this error, you'll need to investigate the specific circumstances and causes that led to the task not executing or the worker being evicted. Reviewing cluster and pod logs, resource configurations, and your DAG definition can help pinpoint the root cause. Adjusting resource requests and limits, configuring tolerations, and setting task priorities are some of the actions you can take to mitigate this issue in a Cloud Composer environment running on Kubernetes.

Posted on --/--/---- --:-- AM
hitjethva81
New Member
Reply posted on --/--/---- --:-- AM

The pod eviction issue in Cloud Composer v2 is often due to resource constraints in the Kubernetes cluster, where nodes may not have enough CPU or memory, leading to the eviction of lower-priority pods. To solve this, you can increase node resources, set higher priorities for critical pods, configure proper resource requests and limits for your pods, and enable node auto-scaling. Additionally, using a Pod Disruption Budget (PDB) can help limit the number of evictions, ensuring important tasks continue running. For more on fixing pod eviction issues related to PDB, check this guide: How to Fix "Cannot Evict Pod as It Would Violate the Pod's Disruption Budget".

0 Likes