Router & Message Processor High Availability

[balakumarm]

We are setting up APIGEE on private cloud (1 DC full HA on AWS). We have some hiccups while testing disaster scenarios.

Problem: trace session not working when we shutdown EC2 (router / MP node) abruptly.

We have two nodes both running Router and Message processor. When we shutdown one node with APIGEE stop script (/opt/apigee/apigee-service/bin/apigee-all stop), we can still trace session from Edge UI and other node serving traffic as normal.

Alternatively, when we shutdown EC2 of one node, trace session in Edge UI not working. Even we tried manually set not reachable for the node, trace session is not working. Getting below exception in Mgmt server.

1. Is there any additional configuration required to make all components work seamlessly (no manual intervention) when minimum required servers running?

2. We notice API traffic is working when EC2 is down. We are not sure is it by chance going to the running node. Is it possible Router on running node forward the traffic to MP on the node not running (EC2 shutdown scenario)?

2018-10-30 04:06:21,299 org:comp env:prod target:/v1/organizations/comp/environments/prod/apis/v-particulars/revisions/3/debugsessions contextId:- action:CREATE qtp1543146693-31 ERROR DISTRIBUTION - DebugSessionAPI.createDebugSession() : createDebugSession : Unable to connect to the server with UUID {}, skipping it1a1994a0-b81f-46a7-97a1-41011a16c60c
org.apache.http.conn.HttpHostConnectException: Connect to 10.1.5.13:8082 [/10.1.5.13] failed: Connection timed out (Connection timed out)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:140) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:115) ~[httpclient-4.3.5.jar:4.3.5]
at com.apigee.distribution.DebugSessionAPI.createDebugSession(DebugSessionAPI.java:242) ~[application-distribution-1.0.0.jar:na]
at sun.reflect.GeneratedMethodAccessor387.invoke(Unknown Source) ~[na:na]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_181]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_181]
at org.apache.cxf.service.invoker.AbstractInvoker.performInvocation(AbstractInvoker.java:173) [cxf-rt-core-2.4.7.jar:2.4.7]
at com.apigee.rest.framework.CustomJAXRSInvoker.performInvocation(CustomJAXRSInvoker.java:126) [rest-1.0.0.jar:na]
at org.apache.cxf.service.invoker.AbstractInvoker.invoke(AbstractInvoker.java:89) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:165) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:241) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
at org.apache.cxf.jaxrs.JAXRSInvoker.invoke(JAXRSInvoker.java:92) [cxf-rt-frontend-jaxrs-2.4.7.jar:2.4.7]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor$1.run(ServiceInvokerInterceptor.java:58) [cxf-rt-core-2.4.7.jar:2.4.7]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_181]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_181]
at org.apache.cxf.workqueue.SynchronousExecutor.execute(SynchronousExecutor.java:37) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.interceptor.ServiceInvokerInterceptor.handleMessage(ServiceInvokerInterceptor.java:106) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.phase.PhaseInterceptorChain.doIntercept(PhaseInterceptorChain.java:263) [cxf-api-2.4.7.jar:2.4.7]
at org.apache.cxf.transport.ChainInitiationObserver.onMessage(ChainInitiationObserver.java:121) [cxf-rt-core-2.4.7.jar:2.4.7]
at org.apache.cxf.transport.http.AbstractHTTPDestination.invoke(AbstractHTTPDestination.java:232) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.ServletController.invokeDestination(ServletController.java:209) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.ServletController.invoke(ServletController.java:152) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.CXFNonSpringServlet.invoke(CXFNonSpringServlet.java:114) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.handleRequest(AbstractHTTPServlet.java:185) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.doPost(AbstractHTTPServlet.java:108) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707) [javax.servlet-api-3.1.0.jar:3.1.0]
at org.apache.cxf.transport.servlet.AbstractHTTPServlet.service(AbstractHTTPServlet.java:164) [cxf-rt-transports-http-2.4.10.jar:2.4.10]
at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:830) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:543) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:463) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548) [jetty-security-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:190) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1584) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:188) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1228) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:168) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:481) [jetty-servlet-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1553) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:166) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1130) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:213) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.Server.handle(Server.java:564) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:318) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:251) [jetty-server-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:279) [jetty-io-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:112) [jetty-io-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:124) [jetty-io-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.Invocable.invokePreferred(Invocable.java:122) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.strategy.ExecutingExecutionStrategy.invoke(ExecutingExecutionStrategy.java:58) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:201) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.run(ExecuteProduceConsume.java:133) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590) [jetty-util-9.4.0.v20161208.jar:9.4.0.v20161208]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_181]
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.PlainSocketImpl.socketConnect(Native Method) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_181]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_181]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_181]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_181]
at org.apache.http.conn.socket.PlainConnectionSocketFactory.connectSocket(PlainConnectionSocketFactory.java:72) ~[httpclient-4.3.5.jar:4.3.5]
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123) ~[httpclient-4.3.5.jar:4.3.5]
... 64 common frames omitted

@Anil Sagar @ Google