restricted_image_training_cpus exhausted error but peak usage at 19%

[ChrisFlowers]

I am running a Vertex AutoML model on GCP (free account). 

I have done 6 attempts/runs, but that they all failed so far. My latest error in the logs says

"The following quota metrics exceed quota limits: aiplatform.googleapis.com/restricted_image_training_cpus"google3.third_party.py.cloud_ml_autoflow.util_lib.error.PublicError: error code: 8".

However, i look at the usage for restricted_image_training_cpus only goes to 8; my quota should allow 42 (default setting). Peak usage in last 7 days is 19,05%. 

So i don't understand why GCP is giving me this error or what i should do.

In tab Configurations: Quota for service CPUs for my location is at 16 out of 24, so 66,67% which seems ok

I was able to reproduce this error and have taken screenshots.

I ve read post below and similar ones, but they don't seem to apply as my threshold do not seem to be hit. https://www.googlecloudcommunity.com/gc/AI-ML/Vertex-AI-tunning-chat-bison-exceeded-quota-exception/...  Anyhow, i am on free account so i can t request quota increase anyhow.

[MJane]

Hi @ChrisFlowers , 

Welcome to Google Cloud Community!

The error message, "The following quota metrics exceed quota limits: aiplatform.googleapis.com/restricted_image_training_cpus," suggests that your usage of CPUs for restricted image training has exceeded the allowed quota. Even though your quota is set to 42, it's possible that the system is enforcing a lower limit or there might be some other factors that could be involved.

Here are workarounds that might help you resolve the error:

Check Quota Limit and Usage - Check your specific quotas by going to the IAM & Admin → qoutas section of the GCP Console. You can also compare your usage across the different regions (if applicable) to ensure that you are not hitting a regional quota limit.

Upgrade to Paid Account or Request Higher Quota - Usage limits for specific tasks, such as image training, might be significantly lower for free-tier accounts compared to paid accounts. Therefore, even if your overall usage seems within the limits, you might still encounter restrictions due to the limitations specific to the free tier.

Review the Configuration - Make sure that you're not selecting a model or configuration that exceeds your quota unintentionally. Make sure you pick a model and setup that works with your usage limits. Check the model type, machine type, and region. Some setups might need extra resources.

If the issue persists, I suggest contacting Google Cloud Support as they can provide more insights to see if the behavior you've encountered is a known issue or specific to your project.

I hope the above information is helpful.

[MaAncheNo]

I'm getting the same error on us-central1: quota at 42 and maximum usage is 8.

"Even though your quota is set to 42, it's possible that the system is enforcing a lower limit or there might be some other factors that could be involved" ... I think that, given that we pay for every failed execution, we deserve a more deterministic approach.

[ChrisFlowers]

I am using the default region US central1 which is would expect to be sufficient. 

I reviewed the configurations & have rerun several times. Seems like a sales trick of Google to me.  Anyhow, if not too expensive, I'll just pay