No quota of 'custom_model_training_nvidia_*' although account upgraded

[kathli]

Hello dear community,

I try to train a model with the following command (Vertex AI is enabled ), but I met error as following. At first I thought it was because I was using Free Trial, so I upgraded  my account, but still get the same error. According to this official doc(https://cloud.google.com/vertex-ai/docs/quotas#custom-trained_model_quotas ), the quota should not be 0 with my region and GPU type config. I checked my quota as well, it shows the following picture which means I don't have any quota on GPUs for custom model training, this doesn't align with the official doc above. I tried with other region and GPU type combination as well, for example "n1-standard-4"  + "NVIDIA_TESLA_T4" + "us-central1", still the same error. Could anyone help me to make it clear please?

gcloud ai custom-jobs create \

--region=us-central1\

--worker-pool-spec=machine-type=a2-highgpu-1g, accelerator-type=

NVIDIA_TESLA_A100, accelerator-count=1,replica-count=1,container-image-uri=$IMAGE_URI \

...

ERROR: (gcloud.ai.custom-jobs.create) RESOURCE_EXHAUSTED: The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_nvidia_a100_gpus

[cassandramae]

Hi @kathli,

Welcome to Google Cloud Community!

It could be that there's a delay in propagation of your newly upgraded project and/or billing account. With this, please retry creating custom training jobs by running the same command and see if it now works.

If the issue persists, I suggest contacting Google Cloud Support for further investigation of your project and confirm if a quota increase is needed.

Hope this helps.

[kathli]

Hi @cassandramae , 

thank you for your reply! I just checked the quotas and run the same command again, nothing happened in these two days, I still have the same error. I'm sure my account is upgraded successfully as I got an E-Mail with "Thanks for activating your full account." as soon as I upgraded it, that was fast. I commit a quota request as well, and just get an update today that it will take 3 to 5 business days, but I think requesting quota should not be the case as my account is not free trial anymore. Thank you!