All of our GKE (autopilot) clusters have stopped being able to reach Firebase auth

[jp-haus]

We started experiencing our GKE pods not being able to make requests to our Firebase auth host site domain. We attributed but not tied yet, to some Autopilot updates it performed over the past weekend. We tested this on GKE 1.32.2 and 1.31.6 and 1.30.9 and seems not related with GKE version updates but instead a networking issue.

Connectivity tests on our GKE pods show that:

bash-5.2# nslookup <project_id>.firebaseapp.com
Server: 169.254.20.10
Address: 169.254.20.10#53

Non-authoritative answer:
Name: <project_id>.firebaseapp.com
Address: 199.36.158.100

Our DNS seems to be able to resolve correctly. But when we curl a http request 

bash-5.2# curl -v https://<project_id>.firebaseapp.com

* Trying 199.36.158.100:443...

bash-5.2# curl -v 199.36.158.100

* Trying 199.36.158.100:80...

bash-5.2# curl -v 199.36.158.100:443
* Trying 199.36.158.100:443...

It times out.

When we run the same requests from an outside of the cluster machine we get:

❯ curl -v https://<project_id>.firebaseapp.com

* Host <project_id>.firebaseapp.com:443 was resolved.

* IPv6: (none)

* IPv4: 199.36.158.100

* Trying 199.36.158.100:443...

* Connected to <project_id>.firebaseapp.com (199.36.158.100) port 443

* ALPN: curl offers h2,http/1.1

* (304) (OUT), TLS handshake, Client hello (1):

* CAfile: /etc/ssl/cert.pem

* CApath: none

* (304) (IN), TLS handshake, Server hello (2):

* (304) (IN), TLS handshake, Unknown (8):

* (304) (IN), TLS handshake, Certificate (11):

* (304) (IN), TLS handshake, CERT verify (15):

* (304) (IN), TLS handshake, Finished (20):

* (304) (OUT), TLS handshake, Finished (20):

...

* Request completely sent off

...

* Connection #0 to host <project_id>.firebaseapp.com left intact

❯ curl -v 199.36.158.100:443
* Trying 199.36.158.100:443...
* Connected to 199.36.158.100 (199.36.158.100) port 443
> GET / HTTP/1.1
> Host: 199.36.158.100:443
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* Received HTTP/0.9 when not allowed
* Closing connection
curl: (1) Received HTTP/0.9 when not allowed

We have also ran connectivity test to the IP resolved form the DNS and they passed successfully from VMs inside of the cluster VPC. Showing the request going to the internet using our NAT configuration.

We don't see any logs from Firebase that requests are being blocked. But also we don't see the incoming requests when sent from the GKE pods. We do see the ones we trigger from external machines.

[jayeshmahajan]

Your TLS Connection output says

• Your pod can reach Firebase over the internet.
• TLS handshake is not blocked (not a firewall, NAT, or network policy issue).
• The issue might not be connectivity-related anymore.
  
  

  1. Check if the request received the expected response.
  Add -L to follow redirects:
  curl -v -L https://<project_id>.firebaseapp.com 
  2. If API calls are failing in your app, check the HTTP response.

  curl -v -L -X GET https://<project_id>.firebaseapp.com 
  
  Look for any error codes like 4xx, 5xx
  
  3. Check firebase security rules thats blocking a call from GKE cluster. 
  
  

[jp-haus]

Hey @jayeshmahajan thanks for your reply.

The ones you are referring are the ones I triggered from machines outside of the cluster. The first 3 requests were from pods in the cluster. The ones that show the TLS handshake were from my local machine.

I tested back again what you suggested and it still timedout:

bash-5.2# curl -v -L <project_id>.firebaseapp.com
* Trying 199.36.158.100:80...

bash-5.2# curl -v -L https://<project_id>.firebaseapp.com 
* Trying 199.36.158.100:443...

---

I also was able to capture the logs from my NAT - and performing requests to other websites such as amazon.com, facebook.com, and others they get logged but firebase dont.

It seems related with the EgressNATPolicy, since the particular request to firebase is the one never reaching the NAT

[jp-haus]

We recently noticed this problem went away for the initial API request on our firebase site hostname but reappeared with a completely different API request through a different vendor

From a pod in the cluster:

bash-5.2# curl -I -v https://haus-demo.myshopify.com
* Trying 23.227.38.74:443...
* connect to 23.227.38.74 port 443 failed: Connection timed out
* Failed to connect to haus-demo.myshopify.com port 443 after 132958 ms: Couldn't connect to server
* Closing connection 0
curl: (28) Failed to connect to haus-demo.myshopify.com port 443 after 132958 ms: Couldn't connect to server

From a local machine:

❯ curl -I -v https://haus-demo.myshopify.com
* Host haus-demo.myshopify.com:443 was resolved.
* IPv6: (none)
* IPv4: 23.227.38.74
* Trying 23.227.38.74:443...
* Connected to haus-demo.myshopify.com (23.227.38.74) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/cert.pem
* CApath: none
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256 / [blank] / UNDEF
* ALPN: server accepted h2