Cannot increase machine type on AI Platform Predic...

omonao · 06-07-2023 08:05 AM

We are running GPU-based online predictions on our AI Platform.
We are applying auto-scaling, but wanted to improve the machine type because the CPU load is higher than the GPU.

The original machine type we were using was n1-standard-4 and we tried to raise it from there to n1-standard-8 or higher, but we got an error on versions create.
The details of the error are below.
```
"error": {
"code": 429,
"message": "The requested number of n1-standard-8 exceeds the quota limit. Current usage/limit: 0/20, Requested: 24.",
"status": "RESOURCE_EXHAUSTED"
}
```
I thought I could apply to increase the quota, but I don't know which quota it corresponds to.

Otherwise, I tried the following
1. if minNodes=1 in n1-standard-8, I was able to deploy, but could not find the resource that was being used more. 2.
2. we were able to create a new n1-standard-16 (with GPU) in Compute Engine.

kvandres

Good day @omonao,

I highly recommend that you reach out to Google Cloud Support for this inquiry, so that they can investigate your project quotas and project resources. You can use this link: https://cloud.google.com/support

Thank you for being part of Google Cloud Community

omonao

Hi, @kvandres ,

Our project is currently only on basic support, do we have to upgrade our support and inquire?

Thanks!

Roderick

Hi @omonao,

The error message you received indicates that the requested number of n1-standard-8 machine types exceeds the quota limit. The error message specifically states that the current usage/limit is 0/20, and you requested 24 instances.

To resolve this issue, you need to request a quota increase for the n1-standard-8 machine type. Here's how you can do it:

Go to the Google Cloud Console: https://console.cloud.google.com/
Open the Navigation menu and select "IAM & Admin" -> "Quotas".
In the "Quotas" page, select the appropriate project from the dropdown at the top.
Use the filter to search for "n1-standard-8" in the "Metric" field.
You should see the quota limit for the n1-standard-8 machine type. Click on it to view details.
Click the "Edit Quotas" button.
Fill in the required details, such as your contact information and the desired quota limit. Explain that you need to increase the quota for n1-standard-8 machine types to support your GPU-based online predictions on AI Platform.
Submit the quota increase request.

Google Cloud support will review your request and respond accordingly. They will let you know if the quota increase is approved or if there are any additional steps required.

In the meantime, you have a couple of alternatives:

You mentioned that you were able to create a new n1-standard-16 machine type with a GPU in Compute Engine. If this meets your requirements, you can continue using this machine type until the quota increase for n1-standard-8 is approved.
If you set the minimum number of nodes to 1 for n1-standard-8 in AI Platform, you can scale up your instance count, but it may not solve the issue of CPU load being higher than GPU load. You might need to optimize your application or workload to better utilize the GPU resources.

Remember to monitor your usage and resources to ensure they align with the quota limits and requirements of your project.

Hope this helps!

omonao

Hi, @Roderick

Thank you for replying!

I tried to investigate on the quota page, but could not find the target quota by searching for "n1-standard-8" in the Metric field.
The same is true when I search for "n1", or try in a different project.

Is there a problem with the project settings or something else?

Thank you for your help.

paintman06

Hi Omonao,
I am having this same issue. Were you able to get this resolved?

zetaattopsortco

Same issue here.

Cannot increase machine type on AI Platform Prediction