As of now, Dataform does not natively provide a mechanism similar to Airflow's sensors to detect the completion of external upstream source data tasks. However, you can use Airflow's capabilities to achieve this.
In Airflow, you can use the @precondition and @trigger decorators to check if a source table is ready in BigQuery and to schedule the execution of your Dataform workflows. Here's an example:
from airflow import precondition, trigger
@precondition(lambda: bigquery.table('my_source_table').exists())
@trigger('@hourly')
def my_dataform_workflow():
# This workflow will only run if the source table exists in BigQuery and it is triggered hourly.
With this approach, your Dataform workflows will only run when the source data is ready in BigQuery, ensuring that they don't execute when the source data hasn't been fully dumped into BigQuery.
LinkedIn
Twitter
