This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Accept
Reject
Preferences
Log in to ask a question
Get hands-on experience with 20+ free Google Cloud products and $300 in free credit for new customers.
Log in to ask a question

Facing 502 errors after switching to Google Cloud Load Balancer

Posted on 10-17-2023 10:01 PM
sai_de
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

Hello,

Previously we had a setup where Nginx was used as a load balancer. It went as follows:

Angular app in the web (Client) --> Nginx in Compute Engine in public subnet, used as load balancer --> Application Server in private subnet, where nginx is used to route requests to the application port based on the path.

In the nginx load balancer, we were adding headers as follows to mitigate the CORS issue.

```

location /app {
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' *;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Content-Type,platform,ServicePlatformID,versionid';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 200;
}
proxy_pass http://127.0.0.1:3000;
}

```

In the new setup we are using Google Cloud Load Balancer as follows:


Angular app in the web (Client) -->Google Cloud Load Balancer --> Application Server in private subnet, where nginx is used to route requests to the application port based on the path.

So we moved the above nginx configuration from load balancing compute engine to application server compute engine.

When we run the requests, it works fine via Postman but when run via the browser, we are constantly seeing the below error in Google CLB.

```
{
"insertId": "1bg7ytof0jd92",
"jsonPayload": {
"backendTargetProjectNumber": "projects/32826632432423",
"remoteIp": "124.153.49.xxx",
"statusDetails": "backend_connection_closed_before_data_sent_to_client",
"@type": "type.googleapis.com/google.cloud.loadbalancing.type.LoadBalancerLogEntry",
"cacheDecision": [
"RESPONSE_HAS_CONTENT_TYPE",
"CACHE_MODE_USE_ORIGIN_HEADERS"
]
}
```

We tried the below 2 solutions, but to no avail.

1) Set `Timeout` and `Connection Draining Timeout` to the same value(in our case, 300) as per this.

2) Set `keepalive_timeout 650s;` in Nginx as per this.

Since none of these solutions helped, we are now stuck. Would appreciate any help to fix this.

Solved Solved
1 4 2,605
Topic Labels
Jump to Solution
1 ACCEPTED SOLUTION

Posted on --/--/---- --:-- AM
sai_de
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

After lots of experimentation, I was able to resolve the issue.

In the nginx configuration

 
```

location /app {
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' *;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Content-Type,platform,ServicePlatformID,versionid';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 200;
}
proxy_pass http://127.0.0.1:3000;
}

```
 
Replacing return 200; with return 204; fixed the issue. I'm not sure why this is the case as return 200; works for both nginx as well as aws load balancer that we use in other applications, but after lots of trail and error, this is the conclusion I've reached.

View solution in original post

Jump to Solution
4 REPLIES 4

Posted on --/--/---- --:-- AM
VannGuce
Staff
Reply posted on --/--/---- --:-- AM

Hi,

Based on the information you shared specifically in the error message that you are having, I recommend checking this troubleshooting documentation[1] of 5xx errors. From the documentation it will explain the “statusDetails” and “backend_connection_closed_before_data_sent_to_client” error message. There is also a recommendation for setting up the keep alive timeout value.

Also, make sure that you follow the required firewall rule[2] for health check, this is one requirement that you need to set up in your firewall when you use a health check.

[1]https://cloud.google.com/load-balancing/docs/https/troubleshooting-ext-https-lbs#troubleshoot_genera...
[2]https://cloud.google.com/load-balancing/docs/health-checks#firewall_rules



 

Jump to Solution

Posted on --/--/---- --:-- AM
sai_de
Bronze 1
Bronze 1
In response to VannGuce
Reply posted on --/--/---- --:-- AM

Hi VannGuce,

Thanks for your reply.

We have already setup `keepalive_timeout 650s;` as mentioned in the initial post in the second last line.

Also, we are not seeing any issues with health check. It is showing a green checkmark in "Healthy" column for the backend in load balancer details page.

Despite all this, we continue to see the "backend_connection_closed_before_data_sent_to_client" issue. It only happens when sending requests via browser, not when using Postman. It has something to do with CORS and Google CLB but I'm unable to figure it out as the same nginx configuration that I posted in my initial message had no issues when nginx was used as load balancer.

Jump to Solution

Posted on --/--/---- --:-- AM
sai_de
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

After lots of experimentation, I was able to resolve the issue.

In the nginx configuration

 
```

location /app {
if ($request_method = 'OPTIONS') {
add_header 'Access-Control-Allow-Origin' *;
add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
add_header 'Access-Control-Allow-Headers' 'Content-Type,platform,ServicePlatformID,versionid';
add_header 'Access-Control-Max-Age' 1728000;
add_header 'Content-Type' 'text/plain charset=UTF-8';
add_header 'Content-Length' 0;
return 200;
}
proxy_pass http://127.0.0.1:3000;
}

```
 
Replacing return 200; with return 204; fixed the issue. I'm not sure why this is the case as return 200; works for both nginx as well as aws load balancer that we use in other applications, but after lots of trail and error, this is the conclusion I've reached.
Jump to Solution

Posted on --/--/---- --:-- AM
mohdrashid1
Bronze 1
Bronze 1
Reply posted on --/--/---- --:-- AM

How to solve 502 bad gateway in GKE (I used GCE Ingress for LB)

```
RROR 2024-07-17T15:44:21.507751246Z [resource.labels.containerName: webapp-api] ' <h1> HTTP Error 500.30 - ANCM In-Process Start Failure </h1>\r\n' +
ERROR 2024-07-17T15:44:21.507768258Z [resource.labels.containerName: webapp-api] '\r\n' +
ERROR 2024-07-17T15:44:21.507778380Z [resource.labels.containerName: webapp-api] ' <h2> Common solutions to this issue: </h2><ul><li>The application failed to start</li><li>The application started but then stopped</li><li>The application started but threw an exception during startup</li></ul>\r\n' +
ERROR 2024-07-17T15:44:21.507784421Z [resource.labels.containerName: webapp-api] '\r\n' +
ERROR 2024-07-17T15:44:21.507790038Z [resource.labels.containerName: webapp-api] ' \r\n' +
ERROR 2024-07-17T15:44:21.507795816Z [resource.labels.containerName: webapp-api] '\r\n' +
ERROR 2024-07-17T15:44:21.507802540Z [resource.labels.containerName: webapp-api] ' <h2> Troubleshooting steps: </h2>\r\n' +
ERROR 2024-07-17T15:44:21.507812459Z [resource.labels.containerName: webapp-api] ' <ul>\r\n' +
ERROR 2024-07-17T15:44:21.507818392Z [resource.labels.containerName: webapp-api] ' <li> Check the system event log for error messages </li>\r\n' +
ERROR 2024-07-17T15:44:21.507824128Z [resource.labels.containerName: webapp-api] " <li> Enable logging the application process' stdout messages </li>\r\n" +
ERROR 2024-07-17T15:44:21.507829863Z [resource.labels.containerName: webapp-api] ' <li> Attach a debugger to the application process and inspect </li>\r\n' +
_keepAliveTimeout: 0,
insecureHTTPParser: undefined,
keepAliveMsecs: 1000,
ERROR 2024-07-17T15:44:21.506707105Z [resource.labels.containerName: webapp-api] response: {
ERROR 2024-07-17T15:44:21.506713270Z [resource.labels.containerName: webapp-api] status: 500,
ERROR 2024-07-17T15:44:21.506724253Z [resource.labels.containerName: webapp-api] statusText: 'Internal Server Error',
ERROR 2024-07-17T15:44:21.506022537Z [resource.labels.containerName: webapp-api] defaultPort: 443,
ERROR 2024-07-17T15:44:21.506028401Z [resource.labels.containerName: webapp-api] protocol: 'https:',
ERROR 2024-07-17T15:42:20.861101553Z [resource.labels.containerName: webapp-api] }
ERROR 2024-07-17T15:44:21.504950127Z [resource.labels.containerName: webapp-api] Error fetching the API: AxiosError: Request failed with status code 500 at settle (/app/node_modules/axios/dist/node/axios.cjs:1983:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3085:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:4224:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) at async EmailService.readingSpec (/app/dist/notification/email_service.js:418:34) at async EmailService.sendPrivacyReport (/app/dist/notification/email_service.js:469:28) at async SensitiveDataService.processDocument (/app/dist/sensitive_data_service/sensitive_data.service.js:269:29) {
ERROR 2024-07-17T15:44:21.505071022Z [resource.labels.containerName: webapp-api] code: 'ERR_BAD_RESPONSE',
ERROR 2024-07-17T15:44:21.505076608Z [resource.labels.containerName: webapp-api] config: {
ERROR 2024-07-17T15:44:21.505221535Z [resource.labels.containerName: webapp-api] transitional: {
ERROR 2024-07-17T15:44:21.505230123Z [resource.labels.containerName: webapp-api] silentJSONParsing: true,
ERROR 2024-07-17T15:44:21.505235562Z [resource.labels.containerName: webapp-api] forcedJSONParsing: true,
ERROR 2024-07-17T15:44:21.505241112Z [resource.labels.containerName: webapp-api] clarifyTimeoutError: false
ERROR 2024-07-17T15:44:21.505246849Z [resource.labels.containerName: webapp-api] },
ERROR 2024-07-17T15:44:21.505253225Z [resource.labels.containerName: webapp-api] adapter: [ 'xhr', 'http', 'fetch' ],
ERROR 2024-07-17T15:44:21.505259837Z [resource.labels.containerName: webapp-api] transformRequest: [ [Function: transformRequest] ],
ERROR 2024-07-17T15:44:21.505265618Z [resource.labels.containerName: webapp-api] transformResponse: [ [Function: transformResponse] ],
ERROR 2024-07-17T15:44:21.505270953Z [resource.labels.containerName: webapp-api] timeout: 0,
ERROR 2024-07-17T15:42:20.861101553Z [resource.labels.containerName: webapp-api] }
ERROR 2024-07-17T15:39:37.055237807Z [resource.labels.containerName: perfai-engine] },
ERROR 2024-07-17T15:39:37.055243893Z [resource.labels.containerName: perfai-engine] socketPath: undefined,
ERROR 2024-07-17T15:39:37.055249982Z [resource.labels.containerName: perfai-engine] method: 'POST',
ERROR 2024-07-17T15:39:37.055257154Z [resource.labels.containerName: perfai-engine] maxHeaderSize: undefined,
ERROR 2024-07-17T15:39:37.055263321Z [resource.labels.containerName: perfai-engine] insecureHTTPParser: undefined,
ERROR 2024-07-17T15:39:37.055269423Z [resource.labels.containerName: perfai-engine] path: '/api-analyzer/fetch-api-spec',
ERROR 2024-07-17T15:39:37.055275495Z [resource.labels.containerName: perfai-engine] _ended: true,
ERROR 2024-07-17T15:39:37.055282051Z [resource.labels.containerName: perfai-engine] res: IncomingMessage {
ERROR 2024-07-17T15:39:37.055288993Z [resource.labels.containerName: perfai-engine] _readableState: [ReadableState],
ERROR 2024-07-17T15:39:37.055301607Z [resource.labels.containerName: perfai-engine] _events: [Object: null prototype],
ERROR 2024-07-17T15:39:37.055313787Z [resource.labels.containerName: perfai-engine] _eventsCount: 4,
ERROR 2024-07-17T15:39:37.055321059Z [resource.labels.containerName: perfai-engine] _maxListeners: undefined,
ERROR 2024-07-17T15:39:37.055343082Z [resource.labels.containerName: perfai-engine] socket: [TLSSocket],
ERROR 2024-07-17T15:39:37.055349871Z [resource.labels.containerName: perfai-engine] httpVersionMajor: 1,
ERROR 2024-07-17T15:39:37.055356241Z [resource.labels.containerName: perfai-engine] httpVersionMinor: 1,
ERROR 2024-07-17T15:39:37.055362507Z [resource.labels.containerName: perfai-engine] httpVersion: '1.1',
ERROR 2024-07-17T15:39:37.055369587Z [resource.labels.containerName: perfai-engine] complete: true,
ERROR 2024-07-17T15:39:37.055376143Z [resource.labels.containerName: perfai-engine] rawHeaders: [Array],
ERROR 2024-07-17T15:39:37.055387644Z [resource.labels.containerName: perfai-engine] rawTrailers: [],
ERROR 2024-07-17T15:39:37.055399576Z [resource.labels.containerName: perfai-engine] aborted: false,
ERROR 2024-07-17T15:39:37.055407889Z [resource.labels.containerName: perfai-engine] upgrade: false,
ERROR 2024-07-17T15:39:37.055414310Z [resource.labels.containerName: perfai-engine] url: '',
ERROR 2024-07-17T15:39:37.055420523Z [resource.labels.containerName: perfai-engine] method: null,
ERROR 2024-07-17T15:39:37.055426958Z [resource.labels.containerName: perfai-engine] statusCode: 502,
ERROR 2024-07-17T15:39:37.055433121Z [resource.labels.containerName: perfai-engine] statusMessage: 'Bad Gateway',
ERROR 2024-07-17T15:39:37.055439371Z [resource.labels.containerName: perfai-engine] client: [TLSSocket],
ERROR 2024-07-17T15:39:37.055445722Z [resource.labels.containerName: perfai-engine] _consuming: false,
ERROR 2024-07-17T15:39:37.055451849Z [resource.labels.containerName: perfai-engine] _dumped: false,
ERROR 2024-07-17T15:39:37.055458097Z [resource.labels.containerName: perfai-engine] req: [Circular *1],
ERROR 2024-07-17T15:39:37.055465585Z [resource.labels.containerName: perfai-engine] responseUrl: 'https://privacy.ai.dev.perfai.ai/api-analyzer/fetch-api-spec',
ERROR 2024-07-17T15:39:37.055529982Z [resource.labels.containerName: perfai-engine] redirects: [],
ERROR 2024-07-17T15:39:37.049334386Z [resource.labels.containerName: perfai-engine] }
ERROR 2024-07-17T15:39:37.053966315Z [resource.labels.containerName: perfai-engine] Error updating field details with AI service: https://privacy.ai.dev.perfai.ai/api-analyzer/fetch-api-spec AxiosError: Request failed with status code 502 at settle (/app/node_modules/axios/dist/node/axios.cjs:1967:12) at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:3066:11) at IncomingMessage.emit (node:events:525:35) at endReadableNT (node:internal/streams/readable:1358:12) at processTicksAndRejections (node:internal/process/task_queues:83:21) at Axios.request (/app/node_modules/axios/dist/node/axios.cjs:3877:41) at runMicrotasks (<anonymous>) at processTicksAndRejections (node:internal/process/task_queues:96:5) {
ERROR 2024-07-17T15:39:37.054107712Z [resource.labels.containerName: perfai-engine] code: 'ERR_BAD_RESPONSE',
ERROR 2024-07-17T15:39:37.054114006Z [resource.labels.containerName: perfai-engine] config: {
ERROR 2024-07-17T15:39:37.054121026Z [resource.labels.containerName: perfai-engine] transitional: {
ERROR 2024-07-17T15:39:37.054126952Z [resource.labels.containerName: perfai-engine] silentJSONParsing: true,
ERROR 2024-07-17T15:39:37.054132876Z [resource.labels.containerName: perfai-engine] forcedJSONParsing: true,
ERROR 2024-07-17T15:39:37.054138781Z [resource.labels.containerName: perfai-engine] clarifyTimeoutError: false
ERROR 2024-07-17T15:39:37.054145246Z [resource.labels.containerName: perfai-engine] },
ERROR 2024-07-17T15:39:37.054152116Z [resource.labels.containerName: perfai-engine] adapter: [ 'xhr', 'http' ],
ERROR 2024-07-17T15:39:37.054158867Z [resource.labels.containerName: perfai-engine] transformRequest: [ [Function: transformRequest] ],
ERROR 2024-07-17T15:39:37.054165172Z [resource.labels.containerName: perfai-engine] transformResponse: [ [Function: transformResponse] ],
ERROR 2024-07-17T15:39:37.054171281Z [resource.labels.containerName: perfai-engine] timeout: 0,
```

Jump to Solution
Top Labels in this Space
Top Solution Authors
View all