LinkedIn
Twitter
Hi @danielkua,
Welcome to Google Cloud Community!
This error means you've reached the limit for a specific resource, in this case, "custom_model_training_cpus". This is the number of CPU cores you can use for custom model training within your Vertex AI project. In addition, you may refer to this documentation for more information on RESOURCE_EXHAUSTED error.
I recommend verifying your quota by visiting the Quotas and System Limits in IAM and Admin tab in Google Cloud Console and filter with metric: aiplatform.googleapis.com/custom_model_training_cpus.
Confirm the usage and limit. You should see both the usage and the limit for this quota. You have the option to request an increase of quota by clicking the three dots on the right side and selecting Edit quota.
See image below for reference.
It might be that you’re encountering a billing account issue. Double-check that your billing account is active and that you have sufficient funds available.
You may also check your pipeline, specifically your training resources. Review your pipeline's configuration if you have a custom training job defined that requires more CPU than your quota allows. Your training code might be overly resource-intensive or your training data is very large, it could be using up more resources than anticipated.
Check if you have any other long-running custom training jobs that might be consuming a large portion of your CPU quota.
Regularly monitor your CPU quota usage to identify potential bottlenecks.
Lastly, experiment with different machine types. Try different machine types for your training jobs to find the optimal balance between cost and performance.
I hope the above information is helpful.
