Custom model training CPU quotas exceeded error

akucia · 09-15-2023 08:22 AM

Hi!

I'm trying to complete the codelab for intro to Vertex AI pipelines https://codelabs.developers.google.com/vertex-pipelines-intro but my pipeline jobs are always failing to run containers due to the following error:

com.google.cloud.ai.platform.common.errors.AiPlatformException: code=RESOURCE_EXHAUSTED, message=The following quota metrics exceed quota limits: aiplatform.googleapis.com/custom_model_training_cpus, cause=null; Failed to create custom job for the task. Task: Project number: 732304752533, Job id: 5833738217119023104, Task id: 8910537889008320512, Task name: product-name, Task state: DRIVER_SUCCEEDED, Execution name: projects/732304752533/locations/europe-west4/metadataStores/default/executions/5380095872214830989; Failed to create external task or refresh its state. Task:Project number: 732304752533, Job id: 5833738217119023104, Task id: 8910537889008320512, Task name: product-name, Task state: DRIVER_SUCCEEDED, Execution name: projects/732304752533/locations/europe-west4/metadataStores/default/executions/5380095872214830989; Failed to handle the pipeline task. Task: Project number: 732304752533, Job id: 5833738217119023104, Task id: 8910537889008320512, Task name: product-name, Task state: DRIVER_SUCCEEDED, Execution name: projects/732304752533/locations/europe-west4/metadataStores/default/executions/5380095872214830989

It seems that I'm exceeding quota for this metric:

aiplatform.googleapis.com/custom_model_training_cpus

But after checking IAM quotas I see that I have a quota of 1 for all CPU types and usage of 0

This seems to be similar to these two issues:

Other stuff I checked:

I checked my billing account and I see no issues reported there
I tried running the pipeline in us-central1 and it reports the same errors

Is there any other quota I need to enable for Vertex AI pipelines? Thanks in advance!