ERROR BOOTSTRAP - RuntimeConfigurationServiceImpl.dispatchToListeners() :

[samuelucich]

HI,

My Private Cloud (AIO) gateway is broken. Can't access any of the proxies, can't redepoy existing proxies, and can't deploy newly created proxies. In the message processor log i see stuff like this:

2018-01-19 10:18:10,787  Apigee-Main-3 ERROR BOOTSTRAP - RuntimeConfigurationServiceImpl.dispatchToListeners() : RuntimeConfigurationServiceImpl.dispatchToListeners : Error occurred while dispatching the request DeployEvent{organization='acclaim', application='proxy-loopback', applicationRevision='2', deploymentSpec=basepath=null;env=test;server=eb2d5407-4063-4548-a5c4-594515d92c26;, deploymentID=7896d95a-9111-4fc9-8eeb-a58d5bececbd} to com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener@74df76f7
com.apigee.kernel.exceptions.spi.UncheckedException: Received an unknown event with description DELETE Application /organizations/acclaim/apiproxies/proxy-loopback/revisions/2/
        at com.apigee.messaging.configuration.MessageProcessorServiceImpl.configure(MessageProcessorServiceImpl.java:635) ~[message-processor-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.configureMessageProcessorService(MessageProcessorBootstrapListener.java:74) ~[application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.unDeploy(MessageProcessorBootstrapListener.java:44) ~[application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.dispatchToListeners(RuntimeConfigurationServiceImpl.java:613) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.unDeploy(RuntimeConfigurationServiceImpl.java:273) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.undeploy(RuntimeConfigurationRPCService.java:127) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.proto.RuntimeConfig_ConfigRPCService_BlockingSkeleton$2.run(RuntimeConfig_ConfigRPCService_BlockingSkeleton.java:55) [application-bootstrap-1.0.0.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-01-19 10:18:10,800  Apigee-Main-3 INFO  ZOOKEEPER - ZooKeeperServiceImpl.delete() : deleting path recursively /organizations/acclaim/environments/test/apiproxies/proxy-loopback/revisions/2/statuses/eb2d5407-4063-4548-a5c4-594515d92c26/errormessage
2018-01-19 10:18:10,802  Apigee-Main-3 INFO  ZOOKEEPER - ZooKeeperServiceImpl.delete() : deleting path recursively /organizations/acclaim/environments/test/apiproxies/proxy-loopback/revisions/2/statuses/eb2d5407-4063-4548-a5c4-594515d92c26/errorcode
2018-01-19 10:18:10,805  Apigee-Main-3 INFO  ZOOKEEPER - ZooKeeperServiceImpl.delete() : deleting path recursively /organizations/acclaim/environments/test/apiproxies/proxy-loopback/revisions/2/statuses/eb2d5407-4063-4548-a5c4-594515d92c26/stacktrace
2018-01-19 10:24:58,704  Apigee-Timer-1 INFO  METRICSLOGGING - MetricsLogger.log() :  Release id 170814_01 RPM apigee-rpm-1.0.0.1373.b8a4afbaef.1709232000-170814_01
        CACHE.total [queue_remaining=500]
        NIO.0.selectors [currentServers=2]
        NIO.1.selectors [currentAccepted=1 accepted=1 currentServers=2 maximumConnectionsOpened=1]
        NIO.2.selectors [currentServers=2]
        NIO.3.selectors [accepted=4 maximumConnectionsOpened=2 closeSuccessful=4 currentServers=2 timeouts=1]
        NIO.4.selectors [accepted=2 maximumConnectionsOpened=2 currentServers=2 currentAccepted=2]
        NIO.5.selectors [accepted=1 currentServers=2 maximumConnectionsOpened=1 timeouts=1 closeSuccessful=1]
        NIO.6.selectors [currentServers=2]
        NIO.7.selectors [currentServers=2]
        NIO.total [currentAccepted=3 accepted=8 closeSuccessful=5 timeouts=2 maximumConnectionsOpened=6 currentServers=2]
        THREADPOOL.Main.statistics [activeThreadCount=1 completedTaskCount=14 corePoolSize=10 largestPoolSize=10 currentPoolSize=10 maxAllowedPoolSize=100]
        THREADPOOL.Timer.statistics [corePoolSize=10 currentPoolSize=10 activeThreadCount=1 completedTaskCount=1161 maxAllowedPoolSize=2147483647 largestPoolSize=10]
        ZOOKEEPER.configuration [connectInfo=State:CONNECTED Timeout:40000 sessionid:0x1610eea10430001 local:/192.168.40.134:53010 remoteserver:192.168.40.134/192.168.40.134:2181 lastZxid:4489 xid:2305 sent:2424 recv:2429 queuedpkts:0 pendingresp:0 queuedevents:0]
        ZOOKEEPER.counters [operations=2232 successfulOperations=2219]
        ZOOKEEPER.operations [exists=1410 setData=96 create=48 getData=390 getChildren=243 delete=45]
        ZOOKEEPER.pathWatcher [create=29 add=21]
        ZOOKEEPER.statistics [operations=2219]
2018-01-19 10:25:39,298  Apigee-Main-6 INFO  MESSAGING.CONFIGURATION - MessageProcessorServiceImpl.configure() : configuring [DeployEvent{organization='acclaim', application='simple-response', applicationRevision='1', deploymentSpec=basepath=/;env=test;, deploymentID=null}]
2018-01-19 10:25:39,299  Apigee-Main-6 ERROR MESSAGING.CONFIGURATION - MessageProcessorServiceImpl.configure() : error configuring config events [DeployEvent{organization='acclaim', application='simple-response', applicationRevision='1', deploymentSpec=basepath=/;env=test;, deploymentID=null}]
com.apigee.kernel.exceptions.spi.UncheckedException: Received an unknown event with description SYNC Application /organizations/acclaim/apiproxies/simple-response/revisions/1/
        at com.apigee.messaging.configuration.MessageProcessorServiceImpl.configure(MessageProcessorServiceImpl.java:635) ~[message-processor-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.configureMessageProcessorService(MessageProcessorBootstrapListener.java:74) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.deploy(MessageProcessorBootstrapListener.java:32) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.dispatchToListeners(RuntimeConfigurationServiceImpl.java:597) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.deploy(RuntimeConfigurationServiceImpl.java:265) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.deployUsingInfos(RuntimeConfigurationRPCService.java:96) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.deploy(RuntimeConfigurationRPCService.java:76) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.proto.RuntimeConfig_ConfigRPCService_BlockingSkeleton$1.run(RuntimeConfig_ConfigRPCService_BlockingSkeleton.java:38) [application-bootstrap-1.0.0.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-01-19 10:25:39,299  Apigee-Main-6 ERROR MESSAGING.RUNTIME - AbstractConfigurator.rollbackTransaction() : Error in processing the changes :
com.apigee.kernel.exceptions.spi.UncheckedException: Received an unknown event with description SYNC Application /organizations/acclaim/apiproxies/simple-response/revisions/1/
        at com.apigee.messaging.configuration.MessageProcessorServiceImpl.configure(MessageProcessorServiceImpl.java:635) ~[message-processor-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.configureMessageProcessorService(MessageProcessorBootstrapListener.java:74) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.deploy(MessageProcessorBootstrapListener.java:32) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.dispatchToListeners(RuntimeConfigurationServiceImpl.java:597) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.deploy(RuntimeConfigurationServiceImpl.java:265) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.deployUsingInfos(RuntimeConfigurationRPCService.java:96) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.deploy(RuntimeConfigurationRPCService.java:76) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.proto.RuntimeConfig_ConfigRPCService_BlockingSkeleton$1.run(RuntimeConfig_ConfigRPCService_BlockingSkeleton.java:38) [application-bootstrap-1.0.0.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-01-19 10:25:39,299  Apigee-Main-6 ERROR BOOTSTRAP - RuntimeConfigurationServiceImpl.dispatchToListeners() : RuntimeConfigurationServiceImpl.dispatchToListeners : Error occurred while dispatching the request DeployEvent{organization='acclaim', application='simple-response', applicationRevision='1', deploymentSpec=basepath=/;env=test;, deploymentID=null} to com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener@74df76f7
com.apigee.kernel.exceptions.spi.UncheckedException: Received an unknown event with description SYNC Application /organizations/acclaim/apiproxies/simple-response/revisions/1/
        at com.apigee.messaging.configuration.MessageProcessorServiceImpl.configure(MessageProcessorServiceImpl.java:635) ~[message-processor-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.configureMessageProcessorService(MessageProcessorBootstrapListener.java:74) ~[application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.listeners.MessageProcessorBootstrapListener.deploy(MessageProcessorBootstrapListener.java:32) ~[application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.dispatchToListeners(RuntimeConfigurationServiceImpl.java:597) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationServiceImpl.deploy(RuntimeConfigurationServiceImpl.java:265) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.deployUsingInfos(RuntimeConfigurationRPCService.java:96) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.RuntimeConfigurationRPCService.deploy(RuntimeConfigurationRPCService.java:76) [application-bootstrap-1.0.0.jar:na]
        at com.apigee.application.bootstrap.proto.RuntimeConfig_ConfigRPCService_BlockingSkeleton$1.run(RuntimeConfig_ConfigRPCService_BlockingSkeleton.java:38) [application-bootstrap-1.0.0.jar:na]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_151]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_151]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_151]
        at java.lang.Thread.run(Thread.java:748) [na:1.8.0_151]
2018-01-19 10:25:39,308  Apigee-Main-6 INFO  ZOOKEEPER - ZooKeeperServiceImpl.delete() : deleting path recursively /organizations/acclaim/environments/test/apiproxies/simple-response/revisions/1/statuses/eb2d5407-4063-4548-a5c4-594515d92c26/errormessage
2018-01-19 10:25:39,311  Apigee-Main-6 INFO  ZOOKEEPER - ZooKeeperServiceImpl.delete() : deleting path recursively /organizations/acclaim/environments/test/apiproxies/simple-response/revisions/1/statuses/eb2d5407-4063-4548-a5c4-594515d92c26/errorcode
2018-01-19 10:25:39,316  Apigee-Main-6 INFO  ZOOKEEPER - ZooKeeperServiceImpl.delete() : deleting path recursively /organizations/acclaim/environments/test/apiproxies/simple-response/revisions/1/statuses/eb2d5407-4063-4548-a5c4-594515d92c26/stacktrace
[root@server1 logs]#