Backend returns 400, but loadbalancer returns 403 status code

[g_krist]

Hi,

I'm pulling my hair out over something I don't understand. We have an backend server behind a Google Load balancer. The backend returns an 400 code, but the loadbalancers logs it as a 403 status code.

See the logging (I redacted the output with X's) :

{
  "insertId": "xxxxxxx",
  "jsonPayload": {
    "cacheDecision": [
      "RESPONSE_HAS_CONTENT_TYPE",
      "REQUEST_HAS_AUTHORIZATION",
      "CACHE_MODE_USE_ORIGIN_HEADERS"
    ],
    "backendTargetProjectNumber": "projects/xxxxxxx",
    "remoteIp": "x.x.x.x",
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "statusDetails": "response_sent_by_backend"
  },
  "httpRequest": {
    "requestMethod": "POST",
    "requestUrl": "https://xxxxxxxx/xxxxxxx",
    "requestSize": "5429",
    "status": 403,
    "responseSize": "359",
    "userAgent": "Xxxxxxx",
    "remoteIp": "xxxxxxx",
    "latency": "0.482871s"
  },
  "resource": {
    "type": "http_load_balancer",
    "labels": {
      "forwarding_rule_name": "http-lb-xxxxxxxx-1-https-ipv4",
      "project_id": "xxxxxx",
      "url_map_name": "web-map-http-xxxxxxx-1",
      "target_proxy_name": "https-lb-xxxxxxx-1-proxy-ipv4",
      "backend_service_name": "web-backend-service-xxxxxxxx-1",
      "zone": "global"
    }
  },
  "timestamp": "2024-09-25T16:15:31.710694Z",
  "severity": "WARNING",
  "logName": "projects/xxxxxx/logs/requests",
  "trace": "projects/xxxxxxx/traces/xxxxxxxxxxxxxxxxxxxxxxxxxxx",
  "receiveTimestamp": "2024-09-25T16:15:32.369137216Z",
  "spanId": "xxxxxxxxxx"
}

Is this intended behaviour, Google? Because I don't understand what an 403 has todo with an 400. I would rather expect an 500 error or something IF Google thinks it has to convert status codes. 

Am I overlooking something here? 

Best wishes

[princesslyca]

Hi @g_krist,

Welcome to Google Cloud Community!

A configuration error in your load balancer setup could be causing this misinterpretation, review your backend code to ensure it's not accidentally returning a 400 Bad Request instead of a 403 Forbidden response. Reviewing the error logs on your backend server could provide more context about the error causing the 400 response.

Also, one possible cause of a 400/403 error is invalid characters in a custom request header or the request headers did not include a content length or transfer-encoding chunked header. See HTTP 400 response reported by external HTTP(S) load balancer and required_body_but_no_content_length

For further assistance, you may contact Google Cloud Customer Care.

I hope the above information is helpful.

[g_krist]

Hi @princesslyca 

Thanks for the answer, but I have the other way around; the backend server is reporting an 400 status code but the loadbalancer logs its as an 403. I would expect it to log as a 400 also then.

But maybe i'm misunderstanding you 🙂