Error "failed to list the clusters" creating cloud serverless runtime template

[jochen_hegele]

Hello,

I try to create a dataproc cloud serverless runtime template to get a spark environment notebook  in a vertex ai workbench instance.

If click on the "New Runtime Template" in category Dataproc Serverless Notebooks I get directly the error " failed to list the clusters" and i am not able to save the creation dialog.

What is the problem? Error makes not much sense because its a serverless thing. or?

Thanks for help.

Best Regards

[jochen_hegele]

Can someone provide information about this? Thanks

[lsolatorio]

Hi @jochen_hegele, 

Thank you for joining our community.

I know it can be frustrating when things aren't working as we expect it to be, coupled with the lack of information for troubleshooting. Consider following this document ("Use the Dataproc JupyterLab plugin for serverless batch and interactive notebook sessions") as it can guide you in resolving your use case.

To perform this, you will need to enable the Cloud Dataproc API, then create a cluster and lastly enable the Dataproc metastore API.

I hope I was able to provide you with useful insights.

[jochen_hegele]

Hello,

thanks for answer. I tried to follow this guide:

https://cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled and use the plugin with dataproc severless for spark: 

https://cloud.google.com/vertex-ai/docs/workbench/instances/create-dataproc-enabled#serverless-spark

Dataproc api was already active. the dataproc metadata API not, but this helped nothing error is still showing up. 

This plugin should be pre-installed on vertex-ai. A dataproc cluster should not be necessary becaus I would like to use the "Dataproc Serverless Notebooks" --> New Runtime Template icon. So no cluster should be necessary or?  

[jochen_hegele]

I think a solved the problem.  A couple of permission roles solved it for the compute engine service account.  Also the VPC subnet of the default network hast to be configured that the "Private Google Access" Option is set to ON. A clear guideline that these things are necessary if you start your journey from Vertex AI as a strating point would be helpful:-)