This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
For the last couple of days we have been observing intermittent connectivity issues to GKE control plane for some of our clusters. TCP connections timing out, api server returning 504s. Anyone else observing similar issues?
Can you add details to your question? Were there any recent changes on the configuration prior to the intermittent connectivity issues? Can you share the error messages related to TCP connections timing out or API server with 504 errors?
No there has been no recent changes to the configuration prior to the issue. I have also checked if our clusters had any automated upgrades of node pools or the control plane, but there hasn't been any recent upgrade. We have also upgraded the control plane on one of the affected clusters and it didn't help. The issue started on 2023-01-11. There is a lot of details but in short making HTTP requests to the control plane IP address fail intermittently. Sometimes the client receives a 504 and sometimes the TCP connection just times out. This is causing our helm based deployments to be very unreliable. The simplest method to reproduce this issue is to make an HTTP request to https://control_plane_api/api url. We have observed this issue on at least two of our clusters, both located in europe-west1. The issue has been reported in: https://issuetracker.google.com/issues/265100751 Some comments with logs included are restricted, so I am not sure you will be able to see them.
I believe it would be best to contact GCP support about this. It needs to have visibilty and access to your project to isolate and determine what causes the TCP connection timeout and error 504s.
This is what we did, we purchased the Enhanced Support plan and raised a ticket. A bit unfortunate that it will cost us 3% of our monthly spend to just have anyone look at this issue.
LinkedIn
Twitter
