Load Balancer outputs weird latency

[bluage_nakayama]

Hi,

I have a question about ALB and timeouts.

We have a web system that utilizes an Application Load Balancer (ALB) connected to our GKE cluster. We have enabled logs and observed something unusual. We configured the timeout value for the backend service to be 30 seconds, but the ALB log indicates that the latency exceeded 30 seconds, showing a value like "162.824886." How could this happen?

The "httpRequest.status" is 200, so the requests seem to be processed correctly.

Thank you so much!

[VannGuce]

Make sure you configure the timeout correctly from the result of your logs, ALB and backend service timeout have different settings. What I think that actually happens for exceeding the timeout value is if the backend service is having a heavy load because it may take longer to respond to a request even if the status is returned 200, same effect also with performance issues.

Another possible reason is ALB buffers a date for a short period of time which can cause an extent of the latency. There are also various components that you can see in latency, waiting time spent for the connection to be established, transfer of data and time processed on backend server, the result from the logs might also capture these values.

I suggest monitoring the load balancer and also the backend, increasing the timeout value and observing if it will give you the same result.

[bluage_nakayama]

Thank you,

The logs of actual backend service show that they didn't take so long. So I suspect that ALB buffers responses from backends and it measures the time which clients actually took to receive all of the data, so the ALB outputs very long time depending on the connection health of the clients.

What do you think? Do they have actually buffers?

Thanks!