Serverless VPC not working without VM external IP address

[sailor]

I use gcloud compute engine for my db and app engine for my server. If on the CE/VM I enable ephemeral external IP, my server can connect via the INTERNAL network. I can also reach my server externally. If however I configure my CE with INTERNAL ONLY then my server cannot reach the CE, even though the server used the internal network with the CE configured for ephemeral external IP. The server knows nothing about the ephemeral IP address of the CE instance. In summary:

CE with default network (10.156.0.X & ephemeral external IP) -> App Engine server (via 10.156.0.X) = works!

CE with default network (10.156.0.X only) -> App Engine server = doesn't work

I would have thought simply removing the external IP address wouldn't have an effect on the internal network! I am currently using the Serverless VPC connector on my server to access the GCE. Both are part of the same project.

I also performed a connectivity test App Engine -> Compute Engine which works fine. I can't understand why the App Engine can't receive the Compute Engine response.

Same/Similar issues, none of which have no solutions to date:

Same issue here:

https://serverfault.com/questions/1106358/google-cloud-serverless-vpc-access-not-working-in-both-dir...

Same issue here:

https://serverfault.com/questions/1102780/cannot-connect-to-a-on-premise-vm-via-cloud-vpn-from-googl...

Same issue here:

https://stackoverflow.com/questions/76298106/google-cloud-run-egress-traffic-to-cloud-vpn

Similar issue here:

https://stackoverflow.com/questions/73883936/is-google-cloud-vpn-only-compatible-with-compute-engine...

[Marramirez]

Hi @sailor,

Welcome to the Google Cloud Community!

You can try Using VPC Firewall Rules.

In VPC firewall rules, you can allow or deny connections to or from your Virtual Machines in your VPC network. Enabled VPC firewall rules are always enforced which protects the instances regardless of their configuration and operating system. You should first take a look at VPC Firewall rules before you use it, as there are implied rules and system-generated rules for default networks. 

Let me know if it helped, thanks!

[sailor]

Hi Marramirex,

It is AppEngine to ComputeEngine (VM). The AE is my middleware (Express server) and the VM is my db. This is seemingly a standard use case for a ServerlessVPN and would have thought there was a 'cookbook' for it. I would like to keep all middleware-to-db communication private and within Google Cloud, which I thought was the purpose of the ServerlessVPN. 

I did perform a connectivity test from AE->CE which passed (even without an external IP on my CE), but it only works with IP 0.0.0.0/0 which seems strange as anything more specific doesn't pass. Also this still doesn't fix my core issue of having a ServerlessVPN without an external IP address.

[knet]

Is this App Engine Standard or Flex? If Standard, I assume you've set up a VPC Connector. I second the suggestion to check the firewall rules. 

[sailor]

Hi Knet

I have a Standard AppEngine. 

I have reviewed the firewall rules and tested them with the Connectivity tester from App to CE. The test only works with rules allowing only the most permissive firewall setting (IP address 0.0.0.0/0, all ports). I could not get the test to work using a rule using the CE & AE service accounts (this should have been easy). It also worked with a more restrictive rule using a CE tag. However if I tried to use an even more restrictive rule such as service-to-service which fails, the previous tagged rule which had worked suddenly stops working and I need to revert to the base least restrictive rule in order for the connectivity test to work again.

As stated in my previous post, even though the connectivity test works, there is still no connection between CE & AE if I remove the external ephemeral IP address on my CE.

In conclusion, I have zero confidence in GCloud:

1. A simple use-case such as mine (middleware server-to-database) using ServerlessVPN doesn't work.

2. There is no proven, documented end-to-end sample implementation such as mine that I can find where this actually does work.

3. I am being charged for connectivity tests (the fact I am being charged is not disclosed; furthermore these tests are not even useful as they don't prove anything).

4. Preventing public access to my database should be a given, instead here in my case I am forced to do just the opposite.

5. No official help from Google is offered.

6.  Their documentation reads like a dictionary with references left, right and center without any detailed explanations of how to tie the elements together. Nor with any proven use cased with all details available upon which to use as a template or benchmark.

7. Suggestions to review my firewall settings, although made in good faith, are akin to saying I should look into a black hole and tell what I see! Sorry, but I'm frustrated. Exactly which rule should I have to make the AE to CE connection without having an external CE IP address?

Regards, Sailor

[knet]

The direction matters here. Are you talking about App Engine -> Compute Engine, or Compute Engine -> App Engine?

If it's Compute Engine -> App Engine, the Compute Engine VM needs to have the ability to reach App Engine's IP address, which is a public IP. I believe this can be achieved either by having the VM have an external IP address, or by enabling Private Google Access. The Cloud Run Private Networking guide may help here; it's a different product, but at least on the ingress path (traffic INTO App Engine), the two products are very similar. https://cloud.google.com/run/docs/securing/private-networking

If you're talking about App Engine -> Compute Engine requests, then I don't know why this behavior would occur.

[sailor]

Hi Knet,

I opened both the ingress and egress firewalls on the CE, but neither solution works if I don't have an external ephemeral IP address open on my CE. Again, what I find odd is that my AE server has no knowledge of the CE db except for the internal IP address even with the external IP address open. So somehow the AE is not accepting the response from the CE (which I doubt given the AE is open to all IP addresses as it never know which client will be calling it) or, the CE response is being blocked or can't use the Severless VPC. 

Again, I haven't found or read or seen a worked, proven example with all the firewall details of how a GCloud db->middleware set up should work in practice. 

Regards, Sailor

[knet]

Hello Sailor, 

Thank you for the additional details. For firewall rules using a service account, it's the service account of the VPC Connector that matters, not the service account of the App Engine service. I'd suggest giving that a try, hopefully that solves the issue. 

This page has some information on how to set up firewall rules for App Engine + VPC connectors: https://cloud.google.com/vpc/docs/configure-serverless-vpc-access 

[knet]

Separately, thank you for the feedback on the documentation. It's helpful to hear where we're falling short; it sounds like you would like to see more end-to-end tutorials.