LinkedIn
Twitter
Background
Apigee Hybrid offers deployment flexibility, allowing you to host the runtime in various environments (e.g., Google Cloud, AWS, on-premises). This flexibility, however, often introduces complexity when it comes to managing ingress traffic. A common pattern is to utilize global or regional load balancers to distribute traffic to the Apigee Hybrid runtime. This document focuses on ALB but is applicable to any Load Balancer.
|
[User] -> [External DNS] -> [AWS Application Load Balancer (ALB)] -> [NLB] -> [Apigee Ingress] |
This high level architecture presents the typical high level scenario of Apigee Global Deployment.
Common Scenarios:
- AWS ALB/NLB with Apigee Hybrid on EKS: As we discussed earlier, using an AWS ALB in front of an Apigee Hybrid runtime on EKS often involves an NLB to expose the Apigee Ingress Gateway. This setup requires careful consideration of hostname-based routing and health checks.
- Google Cloud GLB with Apigee Hybrid: When Apigee Hybrid is deployed in Google Cloud, a Google Cloud Global Load Balancer (GLB) is a natural choice. However, even with GLB, you need to ensure that traffic is correctly routed to the Apigee Ingress and that hostname-based routing is preserved.
- Hybrid Cloud with Multi-Region/Multi-Provider: In the most complex scenarios, Apigee Hybrid might be deployed across multiple regions or even cloud providers. In these cases, a global load balancer is crucial for distributing traffic, but the challenges of hostname-based routing and backend connectivity are amplified.
Key Challenges:
- Hostname-Based Routing: Apigee Hybrid relies on hostname-based routing to direct traffic to the correct API proxy. Traditional load balancers might primarily focus on IP addresses, ports, or paths. AWS ALBs don't support hostname-based target groups. ALBs primarily route based on the path or query parameters, not the Host header.
- Health Checks:Health checks are critical for load balancers to ensure traffic is directed to healthy backend instances. However, configuring health checks that accurately reflect the health of the Apigee Ingress and the underlying API proxies can be particularly challenging, especially when hostname-based routing is a core requirement. In a typical scenario, AWS Application Load Balancer (ALB) is configured to forward traffic to the Apigee Ingress Gateway's Network Load Balancer (NLB) IPs as target groups. This IP-based routing introduces a significant problem: the ALB's health checks against these targets consistently fail, preventing the ALB from routing traffic to the Apigee Ingress Gateway. The root cause of these health check failures lies in the mismatch between the ALB's health check capabilities and Apigee Hybrid's reliance on hostname-based routing. ALBs primarily perform health checks based on IP address, port, and path, without inherently incorporating the Host header. Because Apigee Ingress requires the correct Host header to properly route incoming requests, a simple IP-based health check is insufficient to validate its health and routing capability for specific hostnames. Consequently, the ALB incorrectly perceives the targets as unhealthy, despite the underlying infrastructure potentially being partially operational.
Solution Approach
The solution approach described in the provided text, broken down into two main parts:
Part 1: Health Check Resolution (User-Agent Configuration)
- Problem: The load balancer's health checks to the Apigee Ingress Gateway fails because of the User-Agent Header.
- Root Cause: The Apigee Ingress Gateway's health check mechanism requires a User-Agent header. The load balancer's default health checks may not include this header.
- Solution:
- Configure the Apigee Ingress Gateway to ignore the User-Agent header requirement for health checks.
- This is achieved by modifying the overrides.yaml file to include istiod: healthCheckUserAgents
|
|
- Upgrade the apigee-virtualhost Helm chart with the modified overrides.yaml file.
|
helm upgrade $ENV_GROUP apigee-virtualhost/ --namespace apigee --atomic --set envgroup=$ENV_GROUP -f overrides.yaml |
- Restart the Apigee Ingress Gateway deployment.
|
kubectl rollout restart deployment -n apigee apigee-ingressgateway |
- Verify that the health check now works without the User-Agent header.
|
curl https://nlb-ip/healthz/ingress -v |
Part 2: Routing with Non-SNI Clients (IP based Routing)
- Problem: The AWS Application Load Balancer (ALB) uses IP addresses for target groups, which conflicts with Apigee's hostname-based routing requirements.
- Root Cause: ALBs route traffic based on IP addresses, while Apigee Hybrid needs the Host header to route requests to the correct API proxies.
- Solution:
- Configure an Apigee route with a wildcard hostname (*) to handle requests from non-SNI clients (clients that don't provide the hostname in the TLS handshake). Refer the Apigee documentation here.
- Update the overrides.yaml file to include an additionalGateways configuration for the virtualhosts section. This configuration references the custom route created in the previous step.
|
virtualhosts: |
- Upgrade the apigee-virtualhost Helm chart with the updated overrides.yaml file.
|
helm upgrade $ENV_GROUP apigee-virtualhost/ --namespace apigee --atomic --set envgroup=$ENV_GROUP -f overrides.yaml |
- Verify health check from ALB
At this point, the ALB should show the health check as active and you should be able to call the API with https://alb-hostname/<endoint> assuming alb-hostname is added as host alias in the environment group.
Part 3 (Optional): Custom health Check
In Apigee, you can also do an environment level health check as mentioned in the document here. However with ALB setup as mentioned above custom health checks will fail and require additional steps .
- Navigate to the Apigee Ingress Manager Templates Directory:
cd [HELM_BASE_DIR]/apigee-ingress-manager/templates/
- Modify istiod-envoyfilters.yaml:
Open the istiod-envoyfilters.yaml file.
Add the following line after request_handle:headers():add("x-apigee-heartbeat", "true") as described in point C
|
request_handle:headers():replace("HOST", "non-sni.example.com") |
Note: non-sni.example.com can be any arbitrary hostname and doesn't require a DNS entry.
- Update at Both HTTP and Network Filters sections:
You will need to add the line in two sections:
HTTP_FILTER section (roughly around line 161)
NETWORK_FILTER section (roughly around line 375)
|
if p:sub(1, 8) == "/healthz" then |
- Update Host Alias in Environment Group:
- Go to the Environment Group section in the Cloud Console.
- Update the HostAlias field to match the non-sni.example.com entry you added above.
- Deploy the Changes via Helm:
Run the following commands in the terminal where you manage Helm updates:
|
helm delete ingress-manager -n apigee |
- Verify the Changes:
Once the changes are applied, you can test the configuration by accessing a proxy (e.g., mocktarget) with the /healthz prefix in the URL:
|
curl https://<ntlb-ip>/healthz/mocktarget -k -v |
Conclusion
The document provides a comprehensive guide to addressing the challenges of integrating Apigee Hybrid with AWS Application Load Balancers (ALBs). It highlights the complexities arising from Apigee's hostname-based routing requirements and the limitations of ALB health checks. The document offers practical solutions, including workarounds for health check issues and hostname routing discrepancies. Additionally, it provides guidance on implementing custom health checks for more granular monitoring. By following the steps outlined in this document, users can effectively configure an ALB to work with Apigee Hybrid, ensuring proper traffic routing and health monitoring.