Unable to communicate with HTTPs GCP endpoints using static routes

[mikenorgate]

I'm experiencing a strange problem when attempting to communicate with google services over network peering.

Let me explain a bit more about the current setup. I have a VPC which contains a bunch of VMs, this VPC has private service connect setup for access to google services via internal IPs. One of the VMs is hosting a Wireguard VPN server for connection to external machines, and I've setup a static route in the VPC for the subnet used by Wireguard clients.

All the VMs and Wireguard clients can communicate with each other in both directions.

The problem:

The Wireguard clients are setup to use a DNS server inside the VPC, as such they resolve Google services to their internal IP address, when making a HTTPS request to these addresses it just times out, however making a HTTP request succeeds. Output from curl below

HTTP request:

> curl.exe -vvv http://storage.googleapis.com
*   Trying 10.2.40.5:80...
* Connected to storage.googleapis.com (10.2.40.5) port 80 (#0)
> GET / HTTP/1.1
> Host: storage.googleapis.com
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< X-GUploader-UploadID: ADPycduRxmaoRWUbXNUav843ZSEm8lCuu9jO5P1H7sTJx7SZn0EjKuqh12wwfnJ7cXjZopI4g5ab2cdgJw68H12czZB-vQ
< Content-Type: application/xml; charset=UTF-8
< Content-Length: 181
< Date: Mon, 24 Jul 2023 15:28:08 GMT
< Expires: Mon, 24 Jul 2023 15:28:08 GMT
< Cache-Control: private, max-age=0
< Server: UploadServer
<
<?xml version='1.0' encoding='UTF-8'?><Error><Code>MissingSecurityHeader</Code><Message>Your request was missing a required header.</Message><Details>Authorization</Details></Error>* Connection #0 to host storage.googleapis.com left intact

HTTPS Request

> curl.exe -vvv https://storage.googleapis.com
*   Trying 10.2.40.5:443...
* Connected to storage.googleapis.com (10.2.40.5) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* Recv failure: Connection was reset
* schannel: failed to receive handshake, SSL/TLS connection failed
* Closing connection 0
* schannel: shutting down SSL/TLS connection with storage.googleapis.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
curl: (35) Recv failure: Connection was reset
PS C:\>
* schannel: shutting down SSL/TLS connection with storage.googleapis.com port 443
* Send failure: Connection was reset
* schannel: failed to send close msg: Failed sending data to the peer (bytes written: -1)
curl: (35) Recv failure: Connection was reset

As you can see from these logs both requests are going to the same internal IP (10.2.40.5)