I am trying GKERunStartPodOperator to launch pods on existing GKE cluster user deferrable mode=true and do xcom=true. The operator is able to launch the pods, container is getting executed but the
1. base container is complet
2.side care container continues to run
and because of #2 or I don't, the task is not able come out and on its own as expected in durable mode.
Args Used:
self.namespace=dag_config.get("gke_name_space")
self.cluster_name=dag_config.get("gke_cluster_name")
self.location=dag_config.get("region_name")
self.project_id=dag_config.get("project_id")
self.name=pod_name
self.image=spec["containers"][0]["image"]
self.full_pod_spec=deserialized_pod_spec
self.in_cluster=False
self.get_logs=True
self.do_xcom_push=True
self.deferrable=True
# self.is_delete_operator_pod=True
self.on_finish_action="delete_pod"
self.random_name_suffix=False
self.poll_interval=2
self.termination_grace_period=0
Not able to figure out why this is happing? Any pointer or help on this. any settings that are must when deferable=True? Do we need to define any triggers or sensor. My cloud composer has triggered running.
Log Snippet
[2023-11-15, 02:55:48 UTC] {credentials_provider.py:353} INFO - Getting connection using `google.auth.default()` since no explicit credentials are provided. [2023-11-15, 02:56:18 UTC] {taskinstance.py:1392} INFO - Pausing task as DEFERRED. dag_id=dpdf-pt-dev-dag-CloudflowGKEPodOperator, task_id=launch_gke_pods, execution_date=20231115T025458, start_date=20231115T025546 [2023-11-15, 02:56:18 UTC] {local_task_job.py:212} INFO - Task exited with return code 0 [2023-11-15, 02:56:18 UTC] {taskinstance.py:2599} INFO - 0 downstream tasks scheduled from follow-on schedule check
--------------------------------------------------------------------------------
[2023-11-15, 02:56:33 UTC] {taskinstance.py:1290} INFO - Starting attempt 1 of 1
[2023-11-15, 02:56:33 UTC] {taskinstance.py:1291} INFO -
--------------------------------------------------------------------------------
[2023-11-15, 02:56:33 UTC] {taskinstance.py:1310} INFO - Executing <Task(XXXXXXXXXGKEStartPodOperator): launch_gke_pods> on 2023-11-15 02:54:58+00:00
[2023-11-15, 02:56:33 UTC] {standard_task_runner.py:55} INFO - Started process 64561 to run task
[2023-11-15, 02:56:33 UTC] {standard_task_runner.py:82} INFO - Running: ['airflow', 'tasks', 'run', 'dpdf-pt-dev-dag-XXXXXXXXXGKEPodOperator', 'launch_gke_pods', 'manual__2023-11-15T02:54:58+00:00', '--job-id', '31440', '--raw', '--subdir', 'DAGS_FOLDER/dev/rst/qqqq-pp-dev-dag-XXXXXXXXXGKEPodOperator.py', '--cfg-path', '/tmp/tmps5mev5lk']
[2023-11-15, 02:56:33 UTC] {standard_task_runner.py:83} INFO - Job 31440: Subtask launch_gke_pods
[2023-11-15, 02:56:33 UTC] {task_command.py:393} INFO - Running <TaskInstance: dpdf-pt-dev-dag-XXXXXXXXXGKEPodOperator.launch_gke_pods manual__2023-11-15T02:54:58+00:00 [running]> on host airflow-worker-whsmn
[2023-11-15, 02:56:34 UTC] {base.py:73} INFO - Using connection ID 'google_cloud_default' for task execution.
[2023-11-15, 02:56:34 UTC] {credentials_provider.py:353} INFO - Getting connection using `google.auth.default()` since no explicit credentials are provided.
[2023-11-15, 02:56:34 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:36 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:38 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:40 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:42 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:44 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:46 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:48 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:50 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:52 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:54 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:56 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:56:58 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
[2023-11-15, 02:57:00 UTC] {pod_manager.py:516} INFO - Pod abc-xyzuvt-pod-p6yczain6wi8 has phase Running
LinkedIn
Twitter
