Best option to synchronize DAGs in Airflow

[caracena]

Hello,

I'm new to Airflow and am creating some DAGs that will be chained (serialized) using the TriggerDagRunOperator at the end of each one. The goal is to have ETL processes run sequentially so there's no competition for slots.

Most ETLs can be triggered this way, but some need to run at specific times. For example, some must run every hour (or close to it), every 3 hours, and so on.

I understand that Airflow has its own database, and I was considering creating a parametric table in it where each task inside the chained DAGs could set a flag to "running." This would allow the standalone DAGs to wait until a task ends. Then, the standalone DAGs would set their own flag to "custom-running" (or something similar) so the next task in the chain waits for it to complete. Does that make sense?

Is this feasible? Or is there another way to achieve something like this? To put it in a graphic:

Thanks in advance!

[ms4446]

Using a custom state table and flags introduces unnecessary complexity. While technically feasible, this approach requires careful synchronization and is challenging to maintain over time.

Airflow offers built-in features that simplify workflow management. The TriggerDagRunOperator facilitates chaining DAGs and passing contextual information, allowing downstream DAGs to dynamically determine dependencies. The wait_for_downstream parameter enhances control by pausing execution until downstream tasks are complete. Similarly, the ExternalTaskSensor enables downstream DAGs to wait for specific tasks or entire upstream DAGs to finish, leveraging Airflow’s robust dependency management capabilities.

For strict sequential execution, Airflow Pools can limit task concurrency across DAGs, ensuring only one task runs at a time. TaskGroups provide a modern way to organize related tasks within a single DAG, replacing the deprecated SubDAGs feature. For complex or loosely coupled workflows, external systems like Pub/Sub can trigger DAGs, offering greater flexibility and scalability.

Airflow's built-in tools provide more maintainable, reliable, and scalable solutions compared to a custom state table. For complex workflows, external event-driven mechanisms add additional flexibility. The best approach depends on your specific ETL process requirements, dependency complexity, and maintainability goals.

[caracena]

Thanks for the answer.

The problem I have with sensors is that I won't be able to specify the minimum time a secondary task will run. For example, if I need the standalone DAG to run whenever is possible only after 3 AM (because data updates in the Datalake or Warehouse) but before 4 AM, there's no way (that I know of) to know when it will be triggered as the main DAG can run faster or slower on different days, depending on different external conditions (i.e. slots usage).

Maybe I am wrong and there could be a condition besides the external sensor (some if condition to test the time after each specified task ends). Is there?

Thanks again.

[ms4446 · Accepted solution]

While Airflow's ExternalTaskSensor is effective for tracking task or DAG completion, enforcing specific time constraints with sensors alone can be challenging.

A practical solution is to use a PythonSensor with time-based logic. This sensor can first verify the completion of upstream tasks or DAGs and then check if the current time falls within the desired window. If the time is outside the window, the sensor can pause execution using the reschedule mode until the condition is met, optimizing resource usage.

Another approach involves the TriggerDagRunOperator to chain DAGs. Time constraints can be enforced within the initial tasks of downstream DAGs by implementing time checks before proceeding. Additionally, the wait_for_downstream parameter ensures the triggering DAG pauses until the downstream DAG has completed.

For simpler workflows, combining DAG scheduling with triggers can be effective. The standalone DAG can be scheduled to start at a specific time, while the upstream DAG triggers it only after completing upstream tasks. For more complex scenarios, external event-driven systems like Pub/Sub can trigger downstream DAGs while incorporating real-time, time-based conditions.

Combining Airflow’s sensors, triggers, and time-based logic provides flexible and reliable solutions for enforcing time constraints. The optimal approach will depend on the complexity of your workflow and the degree of control required over execution timing.