By "Google Batch" I'm referring to the new service Google launched about a month or so ago.
https://cloud.google.com/batch
I have a Python script which takes a few minutes to execute at the moment. However with the data it will soon be processing in the next few months this execution time will go from minutes to hours. This is why I am not using Cloud Function or Cloud Run to run this script, both of these have a max 60 minute execution time.
Google Batch came about recently and I wanted to explore this as a possible method to achieve what I'm looking for without just using Compute Engine.
However documentation is sparse across the internet and I can't find a method to "trigger" an already created Batch job by using Cloud Scheduler. I've already successfully manually created a batch job which runs my docker image. Now I need something to trigger this batch job 1x a day, thats it. It would be wonderful if Cloud Scheduler could serve this purpose.
I've seen 1 article describing using GCP Workflow to create a a new Batch job on a cron determined by Cloud Scheduler. Issue with this is its creating a new batch job every time, not simply re-running the already existing one. To be honest I can't even re-run an already executed batch job on the GCP website itself so I don't know if its even possible.
https://www.intertec.io/resource/python-script-on-gcp-batch
Lastly, I've even explored the official Google Batch Python library and could not find anywhere in there some built in function which allows me to "call" a previously created batch job and just re-run it.
https://github.com/googleapis/python-batch
LinkedIn
Twitter
