Re: Really difficult to create VMs with T4 GPU

ly08096 · 06-25-2023 11:23 PM

Although I have GPU quota =4 (all region), it's really frustrating to find a region that I can create multiple GPU VMs. "us-central1" was the only region I could start any GPU VM, and all other attempts failed with "not enough resource". Even in "us-central1", the instance group with autoscaling never create >1 VMs, and always got "The zone currently does have not enough resources for your instance group. Please wait for resources to free up or consider moving to another region or zone."

Is there general shortage of T4 GPU across GCP? Or is it something I configured wrong?

Marvin_Lucero

Hi @ly08096,

Based from the error you obtained,

@ly08096 wrote:

"The zone currently does have not enough resources for your instance group. Please wait for resources to free up or consider moving to another region or zone."

This means that the Google Cloud resources you need are not currently available. I recommend deploying and balancing your workload across multiple zones and regions to reduce the likelihood of an outage and have access to multiple resource pools when you need to expand quickly.

You may want to review this documentation, which outlines how to build resilient and scalable architectures on Google Cloud Platform. Also, note that you are currently using Google Cloud on-demand consumption model which does not provide an assurance inobtaining capacity. GCP offers a feature called reservations that provides a very high level of assurance in obtaining capacity on the Google Cloud Platform, you can check this documentation for details on how to use this feature.