Serverless spark job throwing an error while using shared VPC to connect on-prem storage

[jskalasariya]

I am trying to run simple serverless spark(dataproc batch) job which reads object from on-prem ECS with shared VPC. I have open egress firewall in shared vpc to connect on-prem storage but I don't see that firewall rule is getting hit There are very less resources available at the moment since this is a new product GA recently.

Failed to initialize node gdpic-srvls-batch-fxxxx7-cxx6-4xxd-bxx6-6xxxxxx4-m: Timed out waiting for at least 1 worker(s) registered. This is often caused by firewall rules that prevent Spark workers from communicating with the master. Please review your network firewall rules and be sure they allow communication on all ports between all nodes. See https://cloud.google.com/dataproc-serverless/docs/concepts/network for instructions. See output in: gs://gcs-object-xxxx

I tried looking into url provided but couldn't find much details. If I have to setup NAT gateway with shared VPC project with my project how can I do? Has anyone solved this problem already?

[Marvin_Lucero]

Hi @jskalasariya,

Where you following any documentations or guides when you deployed Spark? If yes, can you share the link/s?

Also, if you are going to use NAT gateway with this setup, can you share your what you're trying to achieve?

The error that you have obtained shows because the 2 conditions where not met. First, make sure that subnet communication is allowed on every ports. Next, PGA must be enabled. Once enabled, you are going to use NAT gateway to provide traffic egress for Spark.

[jskalasariya]

I have created NAT gateway VM instance with nic0 which connects to default non routable internal network and nic1 which connects to shared VPC.  Have forwarding rules which send packets to NAT instance and NAT VM connects to shared VPC where I have created firewall for on-premise connection. All working good so far.