Cloud Task not retrying upon Cloud Run Job failures

[tedlai]

Hi all,

I am using Cloud Task to trigger a Cloud Run Job, and I would like it to automatically retry running the job at a later time whenever the job execution fails. However, what I am observing is that Cloud Task does not retry upon Cloud Run Job failures. Is there a trick to make it work?

[Marramirez]

Hello @tedlai,

Welcome to the Google Cloud Community!

You can Set retry parameters. If a task does not complete successfully, then Cloud Tasks will retry the task depending on the parameters you have set. You can specify the maximum number of tries, set a time limit for a retry attempt, and control interval between attempts.

Additionally, you can also set the maximum retries for Cloud Run jobs. The default number of tries for a task to restart in case of a failure is 3.

Let me know if it helped, thanks!

[tedlai]

Thank you @Marramirez. We do have the retry parameters set. However, Cloud Task cannot tell when a Cloud Run Job fails and does not actually retry, despite having set the retry parameters.

[ShifaliG]

Hello @tedlai,

Could you please share the error message while executing the task?

You can setup the notification channel for understanding if the retries are happening or not and you can get the error details in the same.

For setting up the retries, below command can be used to update your existing job:

gcloud run jobs update JOB_NAME --max-retries RETRY

 Also, you can set up the task timeout option:

gcloud run jobs update JOB_NAME --task-timeout TIMEOUT

This will help the task to understand when it has to opt for retry option.

Links for reference:

https://cloud.google.com/run/docs/configuring/task-timeout

https://cloud.google.com/run/docs/configuring/max-retries 

[tedlai]

Hi @ShifaliG ,

To signal a Cloud Run Job failure, I just throw an exception in my code. In the Cloud Run dashboard, I do see my failed jobs as expected. However, Cloud Task would not retry running the failed jobs.

I have considered setting the retry options of the Cloud Run Job itself. However, there is not an option for retrying after X minutes. Keeping the job running and retrying immediately over and over again would be too costly for me.