This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Announcements
This site is in read only until July 22 as we migrate to a new platform; refer to this community post for more details.
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Ingress returns 502 due to backend_response_corrupted

Posted on 11-02-2023 06:42 AM
drichards_lw
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

I am deploying some IBM products into GKE and after tweaking the Service to have a custom health check the Ingress and LoadBalancer all show "green"

However when we go to the ingress we get a 502 and see this in the Access Log

 

 

jsonPayload: {
cacheDecision: [2]
remoteIp: "160.72.73.xx"
@type: "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry"
statusDetails: "backend_response_corrupted"
backendTargetProjectNumber: "projects/730433976446"
}

 

 

Looking for that error we see in the docs

 

 

The HTTP response body sent by the backend has invalid chunked transfer-encoding or is otherwise corrupted.

 

 

If I get onto another Pod I can curl the backing service, and it works with a port-forward.

 

Here is the Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/backends: '{"k8s1-b8659e0a-kube-system-default-http-backend-80-4a59abfa":"HEALTHY","k8s1-b8659e0a-mft-ns-b2bi-app-b2bi-api-frontend-s-3500-dd91b43c":"HEALTHY","k8s1-b8659e0a-mft-ns-b2bi-app-b2bi-asi-frontend-s-3500-5274a9cb":"HEALTHY"}'
    ingress.kubernetes.io/forwarding-rule: k8s2-fr-yx1k9f8i-mft-ns-b2bi-app-b2bi-ingress-m24orzog
    ingress.kubernetes.io/https-forwarding-rule: k8s2-fs-yx1k9f8i-mft-ns-b2bi-app-b2bi-ingress-m24orzog
    ingress.kubernetes.io/https-target-proxy: k8s2-ts-yx1k9f8i-mft-ns-b2bi-app-b2bi-ingress-m24orzog
    ingress.kubernetes.io/ssl-cert: k8s2-cr-yx1k9f8i-b79pzckigwnbruix-6bbb3e6115fe1834,k8s2-cr-yx1k9f8i-b79pzckigwnbruix-2beeb3ac1748d1e0
    ingress.kubernetes.io/static-ip: k8s2-fr-yx1k9f8i-mft-ns-b2bi-app-b2bi-ingress-m24orzog
    ingress.kubernetes.io/target-proxy: k8s2-tp-yx1k9f8i-mft-ns-b2bi-app-b2bi-ingress-m24orzog
    ingress.kubernetes.io/url-map: k8s2-um-yx1k9f8i-mft-ns-b2bi-app-b2bi-ingress-m24orzog
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"gce","meta.helm.sh/release-name":"b2bi-app","meta.helm.sh/release-namespace":"mft-ns"},"labels":{"app.kubernetes.io/component":"ingress","app.kubernetes.io/instance":"b2bi-app","app.kubernetes.io/managed-by":"Helm","app.kubernetes.io/name":"b2bi","helm.sh/chart":"ibm-b2bi-prod","release":"b2bi-app"},"name":"b2bi-app-b2bi-ingress","namespace":"mft-ns"},"spec":{"ingressClassName":"gce","rules":[{"host":"asi-ingress-internal.corelogic.eliassen.net","http":{"paths":[{"backend":{"service":{"name":"b2bi-app-b2bi-asi-frontend-svc","port":{"name":"https"}}},"path":"/","pathType":"Prefix"}]}},{"host":"api-ingress-internal.corelogic.eliassen.net","http":{"paths":[{"backend":{"service":{"name":"b2bi-app-b2bi-api-frontend-svc","port":{"name":"https"}}},"path":"/","pathType":"Prefix"}]}}],"tls":[{"hosts":["asi-ingress-internal.corelogic.eliassen.net"],"secretName":"b2bi-app-b2bi-asi-frontend-svc"},{"hosts":["ac-ingress-internal.corelogic.eliassen.net"],"secretName":"b2bi-app-b2bi-ac-frontend-svc"}]}}
    kubernetes.io/ingress.class: gce
    meta.helm.sh/release-name: b2bi-app
    meta.helm.sh/release-namespace: mft-ns
  creationTimestamp: "2023-11-01T18:06:28Z"
  finalizers:
  - networking.gke.io/ingress-finalizer-V2
  generation: 1
  labels:
    app.kubernetes.io/component: ingress
    app.kubernetes.io/instance: b2bi-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: b2bi
    helm.sh/chart: ibm-b2bi-prod
    release: b2bi-app
  name: b2bi-app-b2bi-ingress
  namespace: mft-ns
  resourceVersion: "9732105"
  uid: 5c877ba8-3671-42e0-ab36-d344ec55b1d1
spec:
  ingressClassName: gce
  rules:
  - host: asi-ingress-internal.corelogic.eliassen.net
    http:
      paths:
      - backend:
          service:
            name: b2bi-app-b2bi-asi-frontend-svc
            port:
              name: https
        path: /
        pathType: Prefix
  - host: api-ingress-internal.corelogic.eliassen.net
    http:
      paths:
      - backend:
          service:
            name: b2bi-app-b2bi-api-frontend-svc
            port:
              name: https
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - asi-ingress-internal.corelogic.eliassen.net
    secretName: b2bi-app-b2bi-asi-frontend-svc
  - hosts:
    - ac-ingress-internal.corelogic.eliassen.net
    secretName: b2bi-app-b2bi-ac-frontend-svc
status:
  loadBalancer:
    ingress:
    - ip: 34.120.112.xxx

Here is the service

 

apiVersion: v1
kind: Service
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"default": "custom-healcheck-api"}'
    cloud.good.com/app-protocols: '{"https": "HTTPS", "http": "HTTP", "soassl": "HTTPS",
      "soa": "HTTP"}'
    cloud.google.com/neg: '{"ingress":true}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"35006":"k8s1-b8659e0a-mft-ns-b2bi-app-b2bi-api-frontend-s-3500-dd91b43c"},"zones":["us-east1-b","us-east1-d"]}'
    meta.helm.sh/release-name: b2bi-app
    meta.helm.sh/release-namespace: mft-ns
  creationTimestamp: "2023-10-31T00:38:53Z"
  labels:
    app.kubernetes.io/component: api-frontend-svc
    app.kubernetes.io/instance: b2bi-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: b2bi
    helm.sh/chart: ibm-b2bi-prod
    release: b2bi-app
  name: b2bi-app-b2bi-api-frontend-svc
  namespace: mft-ns
  resourceVersion: "9714855"
  uid: 3728bbbf-dadf-4b31-b3c1-f05a6675678a
spec:
  clusterIP: 34.118.230.xxx
  clusterIPs:
  - 34.118.230.xxx
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 35005
    protocol: TCP
    targetPort: http
  - name: https
    port: 35006
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: api-server
    app.kubernetes.io/instance: b2bi-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: b2bi
    helm.sh/chart: ibm-b2bi-prod
    release: b2bi-app
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  type: ClusterIP
status:
  loadBalancer: {}

 

 

 

What things should I check? 

0 2 572
Topic Labels
0 Likes
Reply
2 REPLIES 2

Posted on --/--/---- --:-- AM
drichards_lw
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

I have tried to update the backend service but still get the 502

---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: custom-healcheck-api-http
  namespace: mft-ns
spec:
  healthCheck:
    checkIntervalSec: 15
    port: 50083
    type: HTTP
    requestPath: /propertyUI/app
---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: custom-healcheck-api-https
  namespace: mft-ns
spec:
  healthCheck:
    checkIntervalSec: 15
    port: 50084
    type: HTTPS
    requestPath: /propertyUI/app

I then updated the service to use this new policy

apiVersion: v1
kind: Service
metadata:
  annotations:
    beta.cloud.google.com/backend-config: '{"ports": {"https": "custom-healcheck-api-https",
      "http": "custom-healcheck-api-http"}}'
    cloud.good.com/app-protocols: '{"https": "HTTPS", "http": "HTTP"}'
    cloud.google.com/neg: '{"ingress":true}'
    cloud.google.com/neg-status: '{"network_endpoint_groups":{"35006":"k8s1-b8659e0a-mft-ns-b2bi-app-b2bi-api-frontend-s-3500-dd91b43c"},"zones":["us-east1-b","us-east1-d"]}'
    meta.helm.sh/release-name: b2bi-app
    meta.helm.sh/release-namespace: mft-ns
  creationTimestamp: "2023-10-31T00:38:53Z"
  labels:
    app.kubernetes.io/component: api-frontend-svc
    app.kubernetes.io/instance: b2bi-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: b2bi
    helm.sh/chart: ibm-b2bi-prod
    release: b2bi-app
  name: b2bi-app-b2bi-api-frontend-svc
  namespace: mft-ns
  resourceVersion: "10628614"
  uid: 3728bbbf-dadf-4b31-b3c1-f05a6675678a
spec:
  clusterIP: 34.118.230.xxx
  clusterIPs:
  - 34.118.230.xxx
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    port: 35005
    protocol: TCP
    targetPort: http
  - name: https
    port: 35006
    protocol: TCP
    targetPort: https
  selector:
    app.kubernetes.io/component: api-server
    app.kubernetes.io/instance: b2bi-app
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: b2bi
    helm.sh/chart: ibm-b2bi-prod
    release: b2bi-app
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800
  type: ClusterIP

 

 

 

0 Likes
Reply

Posted on --/--/---- --:-- AM
drichards_lw
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

If I change the service to `type: NodePort` then the ingress works!?!?!?!

0 Likes
Reply
Top Labels in this Space