we are planning to migrate pipeline that are running from composer to workflows.
We have a lot of steps like
1.Moving files source bucket to destination bucket.
2.Transforming data from files and loading to big query tables.
3.Triggering cloud run job to send those data to few APIs from bigquery table and update the status of the call to the particular record
4.Archiving the file that we received.
Here we need to schedule the pipeline to run every 20 minutes.
Some times the pipeline may run beyond 20 minutes.
How can we make the pipeline to wait till previous execution is completed.
LinkedIn
Twitter
