This article is an extension of an existing article that details the ways Hybrid Runtime Ingress can be exposed for consumption. This article focuses on the steps that need to be executed for exposing Apigee Hybrid Ingress on HTTPS and HTTP via L7 external load balancer.
This execution is validated on Apigee Hybrid 1.9 deployed in GKE. With minor tweaks the same can be done for other Kubernetes platforms.
Complete Apigee Hybrid Installation as detailed here. Deploy an apigee proxy with base path apigee-hybrid-helloworld via management console
Setup the following variables:
export PROJECT_ID=<Apigee Hybr id GCP Project> |
Service definition to enable services exposed on the GKE nodes.
cat <<EOF | kubectl apply -f - apiVersion: v1 kind: Service metadata: name: $ENV_GROUP-ingrs-service namespace: apigee annotations: cloud.google.com/app-protocols: '{"https":"HTTPS","http2":"HTTP"}' cloud.google.com/backend-config: '{"ports": {"443": "http-hc-config","80": "http-hc-config"}}' cloud.google.com/neg: '{"ingress": true}' spec: ports: - name: status-port port: 15021 protocol: TCP targetPort: 15021 - name: http2 port: 80 protocol: TCP targetPort: 8080 - name: https port: 443 protocol: TCP targetPort: 8443 selector: app: apigee-ingressgateway #required ingress_name: $INGRESS_NAME org: $ORG_NAME type: NodePort EOF |
Create static external IP address which will be assigned to the Load balancer.
gcloud compute addresses create apigee-global --global echo $EXTERNAL_IP_ADDR; |
Create Backendconfig object, this creates the needed Healthcheck for the Load balancer backend services.
cat <<EOF | kubectl apply -f - |
Starting with Apigee Hybrid release 1.9.1 HTTP (Port 80) Ingress is disabled. From security standpoint use of non-secure ingress should be avoided. As per release notes. "Removed port 80 from the default Kubernetes service of Apigee Ingress Gateway."
The Ingress resource will create the external load balancer with the external address created in the above step.
cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.global-static-ip-name: apigee-global name: xlb-http-apigee-ingress namespace: apigee spec: defaultBackend: service: name: $ENV_GROUP-ingrs-service port: number: 80 EOF |
ApigeeRoute definition will enable non-SNI client to reach the Ingress. The loadbalancer healthchecks does not have SNI in the request. This enables Ingress to respond to the healthcheck calls.
cat <<EOF | kubectl apply -f - apiVersion: apigee.cloud.google.com/v1alpha1 kind: ApigeeRoute metadata: name: wildcard-gateway-apigee namespace: apigee spec: hostnames: - "*" ports: - number: 80 protocol: HTTP selector: app: apigee-ingressgateway enableNonSniClient: true EOF |
In the $HYBRID_FILES/overrides/overrides.yaml add the following:
additionalGateways: ["wildcard-gateway-apigee"] under the virtualHosts block under env-group name and run apigeectl apply.
cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml |
Test the endpoint:
(make sure $ENV_GROUP_HOSTNAME value is added as hostname for the Apigee environment group)
curl http://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld --resolve $ENV_GROUP_HOSTNAME:80:$EXTERNAL_IP_ADDR |
A successful response from the above request validates the traffic is reachable from external loadbalancer to the deployed runtime pods and executed.
For the purpose of this article we will use Cert-Manager issuer for Let’s Encrypt and automatically provision certificates on your Apigee hybrid ingress load balancer. Thanks to Daniel Strebel on this article to configure Let’s Encrypt cert provisioning. Some of the steps are modified from the original article to work for version 1.9
Follow this doc to setup a DNS zone. Create record set type A for the hostname (ENV_GROUP_HOSTNAME) that points to your Apigee hybrid ingress IP address (GCP address created in the prerequisites step). Create record set type CAA for the hostname(ENV_GROUP_HOSTNAME) with Routing data value <0 issue "letsencrypt.org"> (just copy the value within the angle brackets).
gcloud config set project $PROJECT_ID gcloud services enable --project=${PROJECT_ID} dns.googleapis.com |
gcloud iam service-accounts create dns01-solver --display-name "dns01-solver" --project $PROJECT_ID gcloud projects add-iam-policy-binding $DNS_PROJECT_ID \ --member serviceAccount:dns01-solver@$PROJECT_ID.iam.gserviceaccount.com \ --role roles/dns.admin gcloud iam service-accounts keys create dns-sa-key.json \ --iam-account dns01-solver@$PROJECT_ID.iam.gserviceaccount.com |
kubectl create secret generic clouddns-dns01-solver-svc-acct \ --from-file=dns-sa-key.json -n apigee |
cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: cloud-dns-issuer namespace: apigee spec: acme: email: $YOUR_EMAIL_ADDRESS server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: name: cloud-dns-issuer-account-key solvers: - dns01: cloudDNS: project: $DNS_PROJECT_ID serviceAccountSecretRef: name: clouddns-dns01-solver-svc-acct key: dns-sa-key.json EOF |
cat <<EOF | kubectl apply -f - apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: cert-manager-$PROJECT_ID-default namespace: apigee spec: secretName: cert-manager-$PROJECT_ID-tls issuerRef: name: cloud-dns-issuer commonName: '*.$ENV_GROUP_HOSTNAME' dnsNames: - $ENV_GROUP_HOSTNAME - '*.$ENV_GROUP_HOSTNAME' EOF |
(wait for 2 mts, the below command should show status as True)
kubectl -n apigee get Certificate cert-manager-$PROJECT_ID-default kubectl get secret cert-manager-$PROJECT_ID-tls -n apigee -o yaml |
If the Certificate Kubernetes object does not have the “Ready” attribute marked as “True” beyond 5 minutes, use the below links as source to troubleshoot the issue:
https://cert-manager.io/docs/troubleshooting/acme/
https://cert-manager.io/v1.6-docs/faq/acme/
https://www.techrepublic.com/article/how-to-add-a-certificate-authority-authorization-record-in-goog...
The Ingress resource will create the external load balancer with the external address created in the above step. (Confirm the service referenced in this Ingress is created as defined in the steps above)
cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.allow-http: "false" kubernetes.io/ingress.global-static-ip-name: apigee-global name: xlb-https-apigee-ingress namespace: apigee spec: defaultBackend: service: name: $ENV_GROUP-ingrs-service port: number: 80 tls: - hosts: - $ENV_GROUP_HOSTNAME secretName: cert-manager-$PROJECT_ID-tls EOF |
Edit ApigeeRoute to indicate the container is enabled to accept traffic on port and to enable non-SNI clients (healthchecks from loadbalancer)
cat <<EOF | kubectl apply -f - apiVersion: apigee.cloud.google.com/v1alpha1 kind: ApigeeRoute metadata: name: wildcard-gateway-apigee namespace: apigee spec: hostnames: - "*" ports: - number: 80 protocol: HTTP selector: app: apigee-ingressgateway enableNonSniClient: true EOF |
In the overrides.yaml add additionalGateways: ["wildcard-gateway-apigee"] under the virtualHosts block under env-group name.
cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml |
Test the endpoint
curl https://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld |
A successful response from the above request validates the traffic is reachable from external loadbalancer to the deployed runtime pods and executed.
If you are using the cert generated by “LetsEncrypt” as detailed in this doc. Use the below to extract the cert and key and initialize the variables.
cd $HYBRID_FILES/certs |
Update the overrides file for the env virtualhost section, the cert and the key file. Replace the sslCertPath and sslKeyPath values as below and run apigeectl apply on the overrides file.
virtualhosts: - name: ENVIRONMENT_GROUP_NAME selector: app: apigee-ingressgateway ingress_name: INGRESS_NAME sslCertPath: $CERT_FILE sslKeyPath: $KEY_FILE |
cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f $HYBRID_FILES/overrides/overrides.yaml |
cat <<EOF | kubectl apply -f - apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations: kubernetes.io/ingress.allow-http: "false" kubernetes.io/ingress.global-static-ip-name: apigee-global name: xlb-https-apigee-ingress namespace: apigee spec: defaultBackend: service: name: $ENV_GROUP-ingrs-service port: number: 443 tls: - hosts: - $ENV_GROUP_HOSTNAME secretName: cert-manager-$PROJECT_ID-tls EOF |
Edit ApigeeRoute “wildcard-gateway-apigee” to contain 443 in the port list. This makes the container to accept traffic from loadbalancer only on port 443.
cat <<EOF | kubectl apply -f - apiVersion: apigee.cloud.google.com/v1alpha1 kind: ApigeeRoute metadata: name: wildcard-gateway-apigee namespace: apigee spec: hostnames: - "*" ports: - number: 443 protocol: HTTPS tls: mode: SIMPLE selector: app: apigee-ingressgateway enableNonSniClient: true EOF |
In the overrides.yaml add additionalGateways: ["wildcard-gateway-apigee"] under the virtualHosts block under env-group name.
cd $HYBRID_FILES ${APIGEECTL_HOME}/apigeectl apply -f overrides/overrides.yaml |
Test the endpoint
curl https://$ENV_GROUP_HOSTNAME/apigee-hybrid-helloworld |
A successful response from the above request validates the traffic is reachable from external loadbalancer to the deployed runtime pods and executed.