Unable to communicate with HTTPs GCP endpoints using static routes

[mikenorgate]

I'm experiencing a strange problem when attempting to communicate with google services over network peering.

Let me explain a bit more about the current setup. I have a VPC which contains a bunch of VMs, this VPC has private service connect setup for access to google services via internal IPs. One of the VMs is hosting a Wireguard VPN server for connection to external machines, and I've setup a static route in the VPC for the subnet used by Wireguard clients.

All the VMs and Wireguard clients can communicate with each other in both directions.

The problem:

The Wireguard clients are setup to use a DNS server inside the VPC, as such they resolve Google services to their internal IP address, when making a HTTPS request to these addresses it just times out, however making a HTTP request succeeds. Output from curl below

HTTP request:

> curl.exe -vvv http://storage.googleapis.com
*   Trying 10.2.40.5:80...
* Connected to storage.googleapis.com (10.2.40.5) port 80 (#0)
> GET / HTTP/1.1
> Host: storage.googleapis.com
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< X-GUploader-UploadID: ADPycduRxmaoRWUbXNUav843ZSEm8lCuu9jO5P1H7sTJx7SZn0EjKuqh12wwfnJ7cXjZopI4g5ab2cdgJw68H12czZB-vQ
< Content-Type: application/xml; charset=UTF-8
< Content-Length: 181
< Date: Mon, 24 Jul 2023 15:28:08 GMT
< Expires: Mon, 24 Jul 2023 15:28:08 GMT
< Cache-Control: private, max-age=0
< Server: UploadServer
<
<?xml version='1.0' encoding='UTF-8'?><Error><Code>MissingSecurityHeader</Code><Message>Your request was missing a required header.</Message><Details>Authorization</Details></Error>* Connection #0 to host storage.googleapis.com left intact

HTTPS Request

> curl.exe -vvv https://storage.googleapis.com
*   Trying 10.2.40.5:443...
* Connected to storage.googleapis.com (10.2.40.5) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* Recv failure: Connection was reset
* schannel: failed to receive handshake, SSL/TLS connection failed
* Closing connection 0
* schannel: shutting down SSL/TLS connection with storage.googleapis.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
curl: (35) Recv failure: Connection was reset
PS C:\>
* schannel: shutting down SSL/TLS connection with storage.googleapis.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
curl: (35) Recv failure: Connection was reset

As you can see from these logs both requests are going to the same internal IP (10.2.40.5)

[Marvin_Lucero]

Hi @mikenorgate ,

It seems that there's a problem with your Wireguard setup when trying to access Google services using HTTPS (secure connections). Here are some simple steps you can follow to troubleshoot and fix the issue:

1. Make sure there are no restrictions in your network that block outgoing HTTPS traffic from the Wireguard clients. Check the settings on both the Wireguard VPN server and the VMs.
2. Sometimes, wrong MTU settings can cause problems with secure connections. Try changing the MTU settings on the Wireguard clients and servers to see if that helps.
3. If you have any SSL inspection or proxy services, check if they are interfering with the connection between Wireguard clients and Google services.
4.  Double-check the DNS settings on the Wireguard clients. Ensure that the DNS server inside the VPC is resolving Google services' names to the right internal IP addresses.

Also, based on the output of the curl command, the issue is likely related to missing or incorrect authorization headers in the HTTPS request made by the Wireguard clients to access Google Cloud Storage services (storage.googleapis.com). You may consider checking the following steps: 

1. Make sure the Wireguard clients include the right authorization details when making requests to Google Cloud Storage. It's like showing the right ID to access a secure area.
2. Check if the Wireguard clients have the correct login information to access Google Cloud Storage. 
3. Make sure that the login information used by the Wireguard clients is still valid and hasn't expired.
4. Confirm that the Wireguard clients are using the secure way (HTTPS) to connect to Google Cloud Storage.
5. Double-check that the Wireguard clients can find the correct address for Google Cloud Storage.

[Andrew_Tyt]

HI @mikenorgate 

I think you have a problem with DNS name resolution, usually, private connected services have a name like *.p.googleapi...  and an SSL certificate approved for this name. Check this document: https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints I suppose it can help 

Best regards

[mikenorgate · Accepted solution]

I managed to get to the bottom of what was happening. It was an MTU issue, due to the MTU of the Wireguard interfaces being smaller than the MTU of the VPC interface it was trying to fragment the packets however the load balancer traffic has the Dont Fragment header set. A few updates to the MTUs across the various machines has got everything working