GCP classic Application Load Balancer encounters backend_timeout 502 solution

[barry_lai]

"jsonPayload": {
    "cacheDecision": [
      "RESPONSE_HAS_CONTENT_TYPE",
      "CACHE_MODE_USE_ORIGIN_HEADERS"
    ],
    "@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
    "statusDetails": "backend_timeout"

"status": 502

 "resource": {
    "type": "http_load_balancer",

You may encounter this problem from time to time

Mainly through a layer of api gateway and then to the api of another service

How to solve this problem

[Manish_B]

Hello @barry_lai,

Thank you for contacting Google Cloud Community. 

For classic Application Load Balancer, an error is generated when the Google Front End(GFE) closes the websocket connection in either idle or active state, after the backend service timeout expires.

Verify that the keepalive configuration parameter for the HTTP server software running on the backend instance is not less than the keepalive timeout of the load balancer, whose value is fixed at 10 minutes (600 seconds) and is not configurable.

The load balancer generates an HTTP 5XX response code when the connection to the backend has unexpectedly closed while sending the HTTP request or before the complete HTTP response has been received. This can happen because the keepalive configuration parameter for the web server software running on the backend instance is less than the fixed keepalive timeout of the load balancer. Ensure that the keepalive timeout configuration for HTTP server software on each backend is set to slightly greater than 10 minutes (the recommended value is 620 seconds).

I hope that above information is helpful 🙂

Thanks & Regards,
Manish Bavireddy.

[tortiz]

Hi @Manish_B 

We have observed the 502 error "backend_connection_closed_before_data_sent_to_client" in the load balancer logs. According to Google's documentation, the load balancer's keepalive timeout is set to 10 minutes (600 seconds) and is not configurable. Following the recommendation, we added the following directive to Apache on the application servers:

• KeepAlive On
• KeepAliveTimeout 620

Server status

However, we noticed that all available threads were occupied, causing slowness and connection errors in the application. Upon reviewing the Apache log, we identified the following warning: "server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting."

The resources (RAM and vCPU) of the application servers were increased, and the MPM module (MaxRequestWorkers, MaxSpareThreads,MinMaxSpareThreads, ServerLimit, TheardsPerChils, StartServers) in Apache was configured according to the server's resources. 

However, when Keepalive is set to 620 seconds on the application servers, they automatically become flooded with waiting connections, degrading the application.

As a result, we have not been able to resolve the 502 error.

Any recommendations that could help us would be greatly appreciated.

[tortiz]

Hi @Manish_B 

We have observed the 502 error "backend_connection_closed_before_data_sent_to_client" in the load balancer logs. According to Google's documentation, the load balancer's keepalive timeout is set to 10 minutes (600 seconds) and is not configurable. Following the recommendation, we added the following directive to Apache on the application servers:

• KeepAlive On
• KeepAliveTimeout 620

Server -status

However, we noticed that all available threads were occupied, causing slowness and connection errors in the application. Upon reviewing the Apache log, we identified the following warning: "server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting."

The resources (RAM and vCPU) of the application servers were increased, and the MPM module (MaxRequestWorkers, MaxSpareThreads,MinMaxSpareThreads, ServerLimit, TheardsPerChils, StartServers) in Apache was configured according to the server's resources. 

However, when Keepalive is set to 620 seconds on the application servers, they automatically become flooded with waiting connections, degrading the application.

As a result, we have not been able to resolve the 502 error.

Any recommendations that could help us would be greatly appreciated.