Workflow and Batch jobs start next step only when previou step succeeds

[gradientopt]

My workflow is the following:

step1 = Branches [Batch Job A.1,Batch Job A.2 ,Batch Job A.3]

step2 = Branches [Batch Job B.1,Batch Job B.2,Batch Job B.3]

I would like step2 to begin ONLY IF all of [Batch Job A.1,Batch Job A.2 ,Batch Job A.3] succeed. Is there a way to enforce that ? 

[Marramirez]

Hello @gradientopt!

Yes, there is a way to enforce that. See Run a Batch Job using Workflows. Workflow allows you to execute the services in an order that you defined using the workflow syntax. In the above documentation, you will be utilizing the Workflows connector for batch to schedule and run a batch job. 

If the above option doesn't work, you can contact Google Cloud Support to further look into your case. Let me know if it helped, thanks!

[gradientopt]

Thanks for the reply! Yeah I am already following Run a Batch Job using Workflows but I am not sure if the result of step1 will affect the execution of step2. Let' s say if one of the branches (which is  Batch Job) in step 1 fails, would step 2 automatically pause? 

[gradientopt]

Hi, workflow has a max timeout of 1800 seconds which is too short for Batch jobs. Is there a way to get around that or an alternative solution? Thanks!

[Shamel]

@gradientopt  This reply is assuming that Job A is a single job and A.1, A.2, A.3, etc. are tasks within job A. A potential option would be to use the IN_ORDER flag within Google Batch to run each task from Job A sequentially in increasing order. So your Workflow will trigger Job A that runs tasks in sequence, and then your next step in the Workflow will run Job B.