Solved: Tasks both in "no state" and in success state in C... - Page 2

tmdedge · 04-24-2024 04:50 AM

Hello,

I am using the managed version of Apache Airflow with Cloud Composer in version composer-2.6.1-airflow-2.6.3.

I have got some of my tasks that are both in a success state (I am able to find them and their logs using Browse > Task Instances) and in no state at all (when going in the grid view of the DAG, there is simply no square representing the task, see screenshot). Tasks are not in the removed state, there is really no square hence no state.

tmdedge

OK, if someone is trying to access the airflow db from the "managed" version of apache airflow by google, here is how I did it.

First of all, you need to find out what the host / user / password / port / name are.

In order to do that, you can export the list of connections using the airflow CLI.

gcloud composer environments run --location=YOUR_REGION YOUR_COMPOSER_ENV_NAME connections export -- - --format=env | grep airflow_db

The connection uri contains all the information you need.

Then, you need to setup kubernetes on your machine.

Find out the namespace in which your composer resources are deployed using

kubectl describe ns

(one of the namespaces is actually kind of the name of your composer version)

Then you need to identify a pod that has access to the airflow db, for instance the name of a scheduler pod:

kubectl get pods --namespace=YOUR_COMPOSER_NAMESPACE

Run a terminal in this pod using

kubectl exec --namespace=YOUR_COMPOSER_NAMESPACE -it YOUR_SCHEDULER_POD /bin/bash

Great, you are logged in the pod. Using psql, you are now able to access the airflow db (using the values retrieved in the first step)

psql -h HOST -U USER -p PORT -d DATABASE_NAME

Enjoy!

View solution in original post

Tasks both in "no state" and in success state in Cloud Composer