Failed to create a vertex AI workbench notebook with g2 machine

[r4ruixi]

I want to create a g2 machine with L4 GPU for vertex AI workbench instance. After I configured everything, i got the following error when creating it in my console.

"Could not create notebook: Invalid value at 'instance.accelerator_config.type' (type.googleapis.com/google.cloud.notebooks.v1.Instance.AcceleratorType), "NVIDIA_L4"

Could anyone help a bit? I just simply selected g2-standard-8 and the configurations are all default.

[kvandres]

Hi @r4ruixi,

I tried to reproduce the issue on my end, but I was able to create an instance with the same machine type g2-standard-8 with default configurations in several zones, you can try changing the region and zone when you are creating an instance I was able to create an instance in the zone us-central1-a but if the issue persists in other zones, I highly recommend that you create an issue in the public issue tracker for this inquiry to contact technical support since this feature is still in preview or you can also send a message to workbench-v2-trusted-tester-external@google.com.
You can check this link for more information: https://cloud.google.com/vertex-ai/docs/workbench/instances/introduction#support

Thank you for being part of Google Cloud Community

[howellzg]

This is not quite the same error, but I got "INVALID_ARGUMENT: Machine type "g2-standard-8" is not supported.". Googling that error led to that page & I believe the argument here of "incorrect region" is correct for my error.

[jonathanttng]

Hi @howellzg ,  I got the same error. Do you have any solution on this subject? I tried g2-standard-16 or g2-standard-32, got the same problem as well. 

[jamesp]

Same problem here. I am guessing that VertexAI has not been updated to "see" the NVIDIA_L4 type, even though they show it in the GUI as an option.

While the GCP staff member mentioned that they were able to launch a machine, I was able to confirm the original poster's inability to do so.

I got a bounceback from the @Google.com email suggested in that post.

[r4ruixi]

Update: this issue depends on the region you choose. If I choose us-west-1 region (tested on zone a), it would fail as i said above; if i choose us-central1 region (tested on zone a), it could provision the machine normally. My hypothesis is that L4 GPU availability also depends on the region, or the deployed Google API version on Google Cloud is region-dependent. 

EDIT:

In fact i did not successfully start an G2 machine on vertex AI in us-central1. Somehow I clicked the G2 machine but the console started an E2 machine for me. 

[jamesp]

Thanks, that's very interesting. Yes I am trying in us-west1, so this fits with your hypothesis. For posterity, I can launch a GCP L4 VM without Vertex AI in us-west1, so it's not just a resource issue.

[r4ruixi]

I could do the same, too. I guess it is probably from Google API's limitation. If you look at the google's API v1, you'll find that L4 GPU is not listed in the notebook API. https://github.com/googleapis/googleapis/blob/master/google/cloud/notebooks/v1/instance.proto#L69-L1...