External load balancer NEG healtcheck failing to GKE pods

[MH1169]

Hi,

I have configured an ingress resource in GKE which creates an external LB to the istio pods in the GKE cluster.

NEG endpoints are correctly added. But healtcheck to the pods itself are failing.

The ingress FW rule to allow 130.211.0.0/22 and 35.191.0.0/16 to port 8443 has been added. Am I missing any configuration?

Thanks.

[glen_yu · Accepted solution]

The (I'm assuming) HTTP load balancer will only health check against HTTP port 80 (plus a handful of other default settings) unless you tell it otherwise.  So while you did put in the correct FW rules for the health check, your health check was never going to port 8443 anyway.  

What you need is a BackendConfig that defines a custom health check port/path/protocol, etc.

Here's an example of a BackendConfig with a custom health check defined.

You'll also need to point your Ingress at it in the annotations.  Example here.

[MH1169]

Hi Glen,

Thanks for the pointers. Have added the configuration and customized backend health check.

Found out that the pod readiness probe (istio service) is failing that caused the NEG readiness probe to fail.

Will check further why istio internal readiness is failing.

Thanks.

[rjtg1992]

I have the same problem in two scenarios, the first one when I scale the deployment in the istio-system namespace istio-ingress-gateway the issue is basically the load balancer create a new exponents making some request fails, do you have a idea about a good solution ?