I was having a similar issue with connecting from Dataproc Serverless to an on-prem storage appliance. Setting egress firewall rules did not work. Tried using a custom container and that also did not work. It is very frustrating that this limitation ...