Automating Batch Jobs in GCP using Terraform and Cloud Scheduler

[harshada2828]

Hi everyone,

I’m trying to create a batch job in Google Cloud Platform (GCP) using Terraform. Based on the official documentation, it seems that creating a batch job directly through Terraform isn't possible. Instead, we need to use Cloud Scheduler to automate the execution of Cloud Batch Jobs.

Here’s my current problem:

1. Goal: I want to create a single batch job, but the Cloud Scheduler is set to repeatedly trigger the batch job, one after another, which is not what I want.

2. Current Approach: I found a solution to pause the scheduler after creating the job. By manually updating the pause variable in Terraform and reapplying, I can stop the scheduler from triggering the job again.

Question: Is there any way to automatically stop the Cloud Scheduler from triggering the batch job after it has been created, without requiring manual intervention (like updating the pause variable)?

Or Is there any other alternatives to create a GCP Batch Job using Terraform.

Any help or insights would be greatly appreciated!

Thanks in advance.

[greb · Accepted solution]

Hi @harshada2828,

Yes you are correct! Creating a one-off job with Cloud Batch, using Terraform is not directly possible. If a Cloud Scheduler is set as a trigger, it will keep executing unless it is stopped manually, and that is not ideal.

Here are a few methods to achieve one-off executions:

1. Set cloud scheduler to be disabled after execution.

• Deploy a Cloud Function that subscribes to Cloud Batch job finished events.
• Once the job is done, the function disables the Cloud Scheduler job with the help of Cloud Scheduler API.
• This eliminates the need for manual changes in Terraform as well.

2. Work with Google Cloud Workflows Instead of Cloud Scheduler.

• Workflows can invoke a Cloud Batch job programmatically with a single invocation and then stop further execution.
• Unlike Cloud Scheduler, workflows do not need to be explicitly told to stop.

3. Use Terraform’s null_resource with gcloud:

• Create a resource ‘null_resource’ and set it to use the local-exec provisioner of Terraform to gcloud batch jobs submit.
• This will make sure that when the Terraform is executed, the null_resource gets evaluated and the job will run only once.

Was this helpful? If so, please accept this answer as “Solution”. If you need additional assistance, reply here within 2 business days and I’ll be happy to help.