Hello,
I'm having two problems when running Dataform ETLs from Airflow. For example, when running an ETL locally from the production release in Datafor that takes 5 minutes, from Airflow it takes 7 or 8 minutes. It always takes from 40% to 60% more time. Am I missing something?
The way I'm doing it is by using 3 tasks/operators per ETL and a task group:
- TaskGroup
- DataformCreateCompilationResultOperator
- DataformCreateWorkflowInvocationOperator
DataformWorkflowInvocationStateSensor
Is there a better (faster) way?
Also, I always get a retry on the DataformWorkflowInvocationStateSensor because a timeout. Any ideas?
LinkedIn
Twitter
