LinkedIn
Twitter
When I was working on migration projects recently, I encountered an unexpected issue related to theGKE (Google Kubernetes Engine) Ingress controller. Specifically, I found that the GKE Ingress controllerdoesn’t support URL path overwriting. Let me explain the issue with an example and walk you through the challenges it caused during my debugging process.
The Scenario
Suppose you have a GKE Ingress controller with different URL path mappings for several services in your cluster. For instance:
- /customer → Maps to the customer service
- /order → Maps to the order service
- /home → Maps to the home service
In a typical setup, when a user makes a request to the Ingress controller, it forwards that request to the correct service. However, rather than forwarding it to the root of the service (e.g., http://service_IP/), the Ingress controller includes the original path in the forwarding request (e.g., http://service_IP/customer).
This means that the application behind each service needs to be aware of the incoming path. For example, if the service receives a request at /customer, it should handle it accordingly. If the application expects the request at the root level (i.e., /), this mismatch becomes problematic.
The Health Check Dilemma
The real trouble arose during the health check configuration. In Kubernetes, each service typically has a health probe configured at the root path (e.g., /health or /).
- Service-Level Health Checks: These health checks are passed as they are configured at the root of the service (e.g., http://service_IP/health).
- Ingress-Level Health Checks: However, when GKE Ingress is involved, the path is included (e.g., http://service_IP/customer/health), causing the health check to fail with a 404 error since the health probe is not found at /customer/health.
Solution
There are two possible solutions that can resolve this issue by either modifying the Ingress behavior or providing a workaround at the service level.
Option 1: Switch to NGINX Ingress Controller
One solution to handle URL path overwriting correctly is to install the NGINX Ingress controller, which supports path rewriting. With path rewriting, the NGINX Ingress controller can remove or modify the path before forwarding the request to the backend service. This way, the service can receive the request at the root path, as expected.
How It Works:
- When a request comes to the Ingress controller, it can rewrite the URL before forwarding it to the backend service.
- For example, a request to http://ingress_IP/customer/health can be rewritten to http://service_IP/health.
This allows your backend services to handle health checks at the root path, without worrying about the path forwarding behavior of the Ingress controller.
Option 2: Install Istio and Use VirtualServices
Another option is to install Istio and define VirtualServices for each service. Istio provides a more advanced service mesh that allows for flexible routing and path forwarding behavior. You can define rules to ensure that when a request comes in with a specific path, Istio can either rewrite the path or direct traffic to the appropriate service in a way that aligns with your service’s expected URL structure.
How It Works:
- With Istio, you can configure VirtualServices that handle path forwarding correctly.
- You can create a rule that modifies the incoming URL path or forwards it to the right service without including the unwanted base path (e.g., removing /customer from the path).
This approach gives you more control over how traffic is routed within your cluster and can be useful if you need more advanced features such as retries, timeouts, or traffic mirroring in addition to path forwarding.
This diagram illustrates the three scenarios:
- Problem with GKE Ingress Controller:
- The user sends a request to /customer/health
- GKE Ingress forwards the request to the Customer Service but keeps the original path: /customer/health
- The service is expecting health checks at /health, resulting in a 404 error
Solution 1: NGINX Ingress Controller:
- The user sends the same request to /customer/health
- NGINX Ingress rewrites the path to /health before forwarding to the Customer Service
- The service receives the request at the expected path and returns a 200 OK
Solution 2: Istio VirtualService:
- The user sends the same request to /customer/health
- Request is routed through Istio Gateway to a VirtualService
- The VirtualService rewrites the path to /health before forwarding to the Customer Service
- The service receives the request at the expected path and returns a 200 OK
Full article : https://medium.com/devops-dev/challenges-with-url-path-forwarding-in-gke-ingress-controller-c175057a...
follow me on Medium : https://medium.com/@rasvihostings
