Batch - Issues failed to start

[JonYu]

When I submitted the GCP Batch task, the batch execution failed and the following log was displayed in the Events logs:

     Job failed when checking quota limits: Batch Error: code - CODE_GCE_SERVICE_UNAVAILABLE, description - googleapi: Error 503: Internal error. Please try again or contact Google Support. (Code: '5FA39F25D5961.A037007.FF0075FA'), backendError.

[islo]

I have the same issue!

[mountainxing]

Hi, 

Thanks for reaching out! There was a service glitch going on while your jobs were submitted. As the issue is gone, would you please give it another try?

Also, would you mind share your opinion on how long you are willing to wait for Batch to keep your job before the job is marked as "failed"? Thanks for your feedback!

[islo]

Is this related to Vertex AI's batch predictions? My batch predictions have been failing, where my last successful run was Apr 25th. I've tried increasing the machine type size

[Shamel]

No, this channel is related to Batch (https://cloud.google.com/batch). The incident below that has affected Batch may also have affected Vertex AI https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY 

[Shamel · Accepted solution]

Hi, this issue has been resolved in Batch as was due to the issue here: https://status.cloud.google.com/incidents/dS9ps52MUnxQfyDGPfkY. Please re-submit your jobs and let us know if you are still seeing this issue. Thank you,

[islo]

I'm still seeing the issue

[Shamel]

Hi @islo - are you seeing the issue in Batch or Vertex AI batch prediction? If it is in Vertex AI batch prediction, and it is the issue you shared above, unfortunately this thread is not specific to that service.

[JonYu]

I have resubmitted it and it is working properly. Thank you

[JonYu]

Hi，I have discovered another issue with GCP Batch. I have enabled access to the shared vpc for the Batch serviceAccount, but when I set up the network and subnet of the shared vpc before submitting, the Batch prompt 403 does not have permission

[mountainxing]

Hi,

We have some document about how to use shared VPC in Batch here: https://cloud.google.com/vpc/docs/shared-vpc, would you please give it a try? Thanks!

[JonYu]

Sorry, I didn't see any information about how batch uses Share VPC in the document you provided. Could you please provide me with a specific URL?

[mountainxing]

Here is the correct URL: https://cloud.google.com/batch/docs/specify-job-network