We were getting Gateway Timeout issue when target backend sends large response.
Target back-end API is working fine without gateway.
So we updated below properties
Router
conf_router_ServerContainer.io.timeout.millis=500000
conf_router_Client.pool.iotimeout=450000
conf_router_Client.pool.connection.timeout=450000
MP
conf_http_HTTPTransport.io.timeout.millis=400000
conf_http_HTTPClient.connect.timeout.millis=20000
We are still getting the same issue but now we have below error in system.log on MP.
Also find the attached zip containing logs, tcpdump, client error screenshot and the trace of the proxy on gateway.
Also we have APIGEE setup with AIO profile.
Please take a look on priority and thanks in advance
2016-07-16 08:47:53,949 org:kronos env:test api:legacy_api-v1-BatchService rev:1 messageid:keng01-dev01-api72-gtw72.dev.mykronos.internal-22125-362-1 NIOThread@0 INFO HTTP.SERVICE - ExceptionHandler.handleException() : Exception java.io.IOException: Broken pipe occurred while writing to channel ChunkedOutputChannel(ClientOutputChannel(ClientChannel[A:10.3.114.25:8998 Remote host:10.3.114.25:41252]@257 useCount=1 bytesRead=0 bytesWritten=1193 age=57410ms lastIO=0ms )) 2016-07-16 08:47:53,950 org:kronos env:test api:legacy_api-v1-BatchService rev:1 messageid:keng01-dev01-api72-gtw72.dev.mykronos.internal-22125-362-1 NIOThread@0 INFO HTTP.SERVICE - ExceptionHandler.handleException() : Exception trace: java.io.IOException: Broken pipe at com.apigee.nio.channels.ClientOutputChannel.writePending(ClientOutputChannel.java:51) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.OutputChannel.onWrite(OutputChannel.java:116) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.OutputChannel.write(OutputChannel.java:81) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.ChunkedOutputChannel.doWritePending(ChunkedOutputChannel.java:43) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.FilterOutputChannel.writePending(FilterOutputChannel.java:37) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.OutputChannel.onWrite(OutputChannel.java:116) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.OutputChannel.write(OutputChannel.java:81) ~[nio-1.0.0.jar:na] at com.apigee.protocol.http.io.BodyPipe.onWrite(BodyPipe.java:169) [http-1.0.0.jar:na] at com.apigee.protocol.http.io.BodyPipe.start(BodyPipe.java:100) [http-1.0.0.jar:na] at com.apigee.protocol.http.io.MessageWriter.onWrite(MessageWriter.java:165) [http-1.0.0.jar:na] at com.apigee.protocol.http.io.MessageWriter.<init>(MessageWriter.java:110) [http-1.0.0.jar:na] at com.apigee.protocol.http.HTTPServer$Context$2.<init>(HTTPServer.java:685) [http-1.0.0.jar:na] at com.apigee.protocol.http.HTTPServer$Context.send(HTTPServer.java:685) [http-1.0.0.jar:na] at com.apigee.messaging.adaptors.http.configuration.MessageProcessorHttpSkeletonFactory$FlowCompletionListener.onFinish(MessageProcessorHttpSkeletonFactory.java:512) [http-adaptor-1.0.0.jar:na] at com.apigee.flow.MessageFlowImpl.onCompletion(MessageFlowImpl.java:213) [message-flow-1.0.0.jar:na] at com.apigee.flow.MessageFlowImpl.resume(MessageFlowImpl.java:175) [message-flow-1.0.0.jar:na] at com.apigee.flow.execution.ExecutionContextImpl$1.run(ExecutionContextImpl.java:93) [message-flow-1.0.0.jar:na] at com.apigee.nio.NIOSelector.runTasks(NIOSelector.java:317) [nio-1.0.0.jar:na] at com.apigee.nio.NIOSelector.access$000(NIOSelector.java:29) [nio-1.0.0.jar:na] at com.apigee.nio.NIOSelector$2.findNext(NIOSelector.java:265) [nio-1.0.0.jar:na] at com.apigee.nio.NIOSelector$2.findNext(NIOSelector.java:256) [nio-1.0.0.jar:na] at com.apigee.nio.util.NonNullIterator.computeNext(NonNullIterator.java:16) [nio-1.0.0.jar:na] at com.apigee.nio.util.AbstractIterator.hasNext(AbstractIterator.java:36) [nio-1.0.0.jar:na] at com.apigee.nio.handlers.NIOThread.run(NIOThread.java:59) [nio-1.0.0.jar:na] Caused by: java.io.IOException: Broken pipe at sun.nio.ch.FileDispatcherImpl.write0(Native Method) ~[na:1.7.0_79] at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47) ~[na:1.7.0_79] at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93) ~[na:1.7.0_79] at sun.nio.ch.IOUtil.write(IOUtil.java:65) ~[na:1.7.0_79] at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:487) ~[na:1.7.0_79] at com.apigee.nio.PlainTransport.write(PlainTransport.java:107) ~[nio-1.0.0.jar:na] at com.apigee.nio.ClientChannel.write(ClientChannel.java:307) ~[nio-1.0.0.jar:na] at com.apigee.nio.channels.ClientOutputChannel.writePending(ClientOutputChannel.java:47) ~[nio-1.0.0.jar:na] ... 23 common frames omitted
Solved! Go to Solution.
Hi,
The problem here is the router timing out. Should be fixed by setting this in /<inst_path>/customer/application/router.properties
conf_load_balancing_load.balancing.driver.proxy.read.timeout=<time in seconds>
Default is 57 seconds, so increase to something higher. The value will be determined by how long you believe a target server should take to respond (+ a little time for the Message processor).
Russell
Hi,
The problem here is the router timing out. Should be fixed by setting this in /<inst_path>/customer/application/router.properties
conf_load_balancing_load.balancing.driver.proxy.read.timeout=<time in seconds>
Default is 57 seconds, so increase to something higher. The value will be determined by how long you believe a target server should take to respond (+ a little time for the Message processor).
Russell
HI @GAURAV
Does this work? I am seeing same broken pipe issue in my MP logs and as a result, it returns HTTP 502. I am wondering if this solves your issue.
This is my router log.
2017-09-06T16:59:58+00:00evpla936.XXXX.comIP:36355IP:89980.700--499-28030POST /shopping/v1/products?brandOnly=true HTTP/1.1-Java/1.7.0apigateway.XXXX.comevpla936.XXXX.com-62682-5742263-5310.68.93.164-
And I have below configuration,
conf_load_balancing_load.balancing.driver.proxy.read.timeout=300 conf_load_balancing_load.balancing.driver.server.keepalive.timeout=300 conf_apigee-nginx-logrotate_pidfile=/opt/apigee/edge-router/nginx/logs/nginx.pid conf_http_HTTPTransport.io.timeout.millis=240000
Thanks,
Krish
I am having the exact same issue, I would love to know if you could finally find the Broken Pipe root cause.
thanks,
Claudio