This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Problem connecting to gcp bigquery from K8s

Posted on 08-15-2023 03:12 AM
deepshikha1
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

We have several pipelines currently using GCP BQ as their destination, in Kubernetes. However, on Friday night and several times before, they failed due to connectivity issues with BQ. Our networks team also had a look to check if any firewall rule might be preventing it from establishing connections, but it looked fine. Can someone please help with finding out the possible root cause for this?

0 1 366
Topic Labels
0 Likes
1 REPLY 1

Posted on --/--/---- --:-- AM
ms4446
Staff
Reply posted on --/--/---- --:-- AM

Possible root causes for connectivity issues between Kubernetes pipelines and GCP BQ:

  • Network configuration issue:There may be a problem with the network configuration between your Kubernetes cluster and GCP BQ. This could be a DNS resolution issue, a VPC Peering issue, or a firewall rule issue.
  • Load balancer issue: If your Kubernetes cluster is using a load balancer, there may be an issue with the load balancer that is preventing traffic from reaching GCP BQ. Make sure that the backend services are healthy if you're using a Network Load Balancer or an HTTP(S) Load Balancer.
  • Pod issue: There may be a problem with one of the pods in your Kubernetes cluster that is preventing it from connecting to GCP BQ. This could be a service account permissions issue, a networking issue, or a container issue.
  • BQ issue: There may be an issue with GCP BQ itself that is preventing it from accepting connections. Check the GCP status page to see if there are any known outages with BigQuery.

Troubleshooting steps:

  1. Check the logs for the pods that are failing to connect to GCP BQ.Use the kubectl logs [POD_NAME] command to see if there are any errors or warnings that can help you identify the issue.
  2. Check the network configuration between your Kubernetes cluster and GCP BQ. Make sure that there are no firewall rules blocking traffic, that the DNS is resolving correctly, and that VPC Peering is set up correctly if necessary.
  3. Check the load balancer, if any,that is used by your Kubernetes cluster. Make sure that it is healthy and that it is not dropping traffic.
  4. Check the pods in your Kubernetes cluster. Make sure that they are healthy and that they have the correct networking configuration.
  5. Check GCP BQ. Make sure that it is healthy and that it is not rejecting connections.

Additional troubleshooting tips:

  • You can use the iperf3 tool to test the connectivity between the pods in your Kubernetes cluster and GCP BQ. This will help you to determine if the issue is with the network or with the pods themselves.
  • You can also use the gcloudcommand-line tool to check the status of GCP BQ. However, it's more accurate to suggest checking the GCP Console or the GCP Status Dashboard, as gcloud doesn't directly provide service health status.