Have similar from some time, restart of management-server and message-processor fix problem for ~300h
My version is 4.15.07.03 (Private Cloud) , each apigee component have on different host,
(on v4.15.04.03 problem exist too)
Error in UI during deploy:
Error in deployment for environment prod.
The revision is deployed, but traffic cannot flow. Call timed out; either server is down or server is not reachable.
/apigee4/var/log/apigee/management-server/logs/system.log:
2016-03-30 02:58:38,587 org:merck env:prod qtp1237989929-96102 INFO DISTRIBUTION - RemoteServicesDeploymentHandler.deploy() : RemoteServicesDeploymentHandler.deploy : Deployment request : DeployEvent{organization='merck', application='up-test', applicationRevision='1', deploymentSpec=basepath=/;env=prod;, deploymentID=null}
2016-03-30 02:58:48,645 org:merck env:prod qtp1237989929-96102 ERROR DISTRIBUTION - RemoteServicesDeploymentHandler.deployToServers() : RemoteServicesDeploymentHandler.deployToServers : Deployment exception for server with uuid 5de18053-0512-4230-9e32-add42c75bfd6 : cause = Call timed out
com.apigee.rpc.RPCException: Call timed out
at com.apigee.rpc.impl.AbstractCallerImpl.handleTimeout(AbstractCallerImpl.java:64) ~[rpc-1.0.0.jar:na]
at com.apigee.rpc.impl.RPCMachineImpl$OutgoingCall.handleTimeout(RPCMachineImpl.java:483) ~[rpc-1.0.0.jar:na]
at com.apigee.rpc.impl.RPCMachineImpl$OutgoingCall.access$000(RPCMachineImpl.java:402) ~[rpc-1.0.0.jar:na]
at com.apigee.rpc.impl.RPCMachineImpl$OutgoingCall$1.run(RPCMachineImpl.java:437) ~[rpc-1.0.0.jar:na]
at io.netty.util.HashedWheelTimer$HashedWheelTimeout.expire(HashedWheelTimer.java:532) ~[netty-all-4.0.0.CR1.jar:na]
at io.netty.util.HashedWheelTimer$Worker.notifyExpiredTimeouts(HashedWheelTimer.java:430) ~[netty-all-4.0.0.CR1.jar:na]
at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:371) ~[netty-all-4.0.0.CR1.jar:na]
at java.lang.Thread.run(Thread.java:745) ~[na:1.7.0_80]
2016-03-30 02:58:48,797 org:merck env:prod qtp1237989929-96102 INFO DISTRIBUTION - RemoteServicesDeploymentHandler.deployToMPsAndRouters() : RemoteServicesDeploymentHandler.deployToMPsAndRouters() - MPs deployment success - deploying to routers
On message-processor:
$ check.sh
Check Processes:
----------------
message-processor (pid 11250) is running...
Check Services:
---------------
Message Processor:
UUID=5de18053-0512-4230-9e32-add42c75bfd6
==> 200
$ curl localhost:8082/v1/servers/self
{
"buildInfo" : {
"buildNumber" : "jenkins-4G_Release_Sanity-1194 ",
"buildTimestamp" : "1449109653537",
"rPMName" : "apigee-rpm-1.0.0.1194.8f421a2.1512030212-1507_01",
"releaseID" : "1507_01",
"sCMBranch" : "origin/OPDK_1507",
"sCMRevision" : "8f421a25aa6c8563a2641d0d5054face9787b627"
},
"externalHostName" : "localhost",
"externalIP" : "xx.xx.xx.xx",
"internalHostName" : "localhost",
"internalIP" : "xx.xx.xx.xx",
"isUp" : true,
"pod" : "gateway",
"reachable" : true,
"region" : "dc-1",
"tags" : {
"property" : [ {
"name" : "jmx.rmi.port",
"value" : "1101"
}, {
"name" : "http.port",
"value" : "8998"
}, {
"name" : "http.management.port",
"value" : "8082"
}, {
"name" : "started.at",
"value" : "1458121818267"
}, {
"name" : "http.ssl.flag",
"value" : "true"
}, {
"name" : "rpc.port",
"value" : "4528"
}, {
"name" : "Profile",
"value" : "MessageProcessor"
}, {
"name" : "up.time",
"value" : "333 hour(s) 56 minute(s) 14 second(s)"
}, {
"name" : "startup.time",
"value" : "2 minute(s) 25 second(s)"
} ]
},
"type" : [ "message-processor" ],
"uUID" : "5de18053-0512-4230-9e32-add42c75bfd6"
}
On management-server
$ check.sh
Check Processes:
----------------
management-server (pid 14471) is running...
Check Services:
---------------
Management Server:
UUID=3c6f5716-d359-4663-953c-c7b85a93cce3
==> 200
Check Registrations:
--------------------
Enter Apigee Edge admin password:
(...)
Next check pod gateway.
--> curl -X GET http://localhost:8080/v1/servers?pod=gateway [...]
[ {
"internalIP" : "xxx.xxx.xxx.xxx",
"isUp" : true,
"pod" : "gateway",
"reachable" : true,
"region" : "dc-1",
"tags" : {
"property" : [ ]
},
"type" : [ "dc-datastore", "cache-datastore", "keyvaluemap-datastore", "kms-datastore", "counter-datastore" ],
"uUID" : "c6d1c0d2-ffa5-4ffe-b950-9b06b75e41eb"
}, {
"externalHostName" : "localhost",
"externalIP" : "xxx.xxx.xxx.xxx",
"internalHostName" : "localhost",
"internalIP" : "xxx.xxx.xxx.xxx",
"isUp" : true,
"pod" : "gateway",
"reachable" : true,
"region" : "dc-1",
"tags" : {
"property" : [ {
"name" : "started.at",
"value" : "1458121713116"
}, {
"name" : "startup.time",
"value" : "14 second(s)"
}, {
"name" : "Profile",
"value" : "Router-Netty"
}, {
"name" : "rpc.port",
"value" : "4527"
}, {
"name" : "http.management.port",
"value" : "8081"
}, {
"name" : "jmx.rmi.port",
"value" : "1100"
}, {
"name" : "up.time",
"value" : "333 hour(s) 54 minute(s) 50 second(s)"
} ]
},
"type" : [ "router" ],
"uUID" : "547ac4d4-8557-42dc-9409-8912833dad0a"
}, {
"internalIP" : "xxx.xxx.xxx.xxx",
"isUp" : true,
"pod" : "gateway",
"reachable" : true,
"region" : "dc-1",
"tags" : {
"property" : [ ]
},
"type" : [ "dc-datastore", "cache-datastore", "keyvaluemap-datastore", "kms-datastore", "counter-datastore" ],
"uUID" : "a55c0c41-e337-4963-ab2e-ce5ac24c05b6"
}, {
"internalIP" : "xxx.xxx.xxx.xxx",
"isUp" : true,
"pod" : "gateway",
"reachable" : true,
"region" : "dc-1",
"tags" : {
"property" : [ ]
},
"type" : [ "dc-datastore", "cache-datastore", "keyvaluemap-datastore", "kms-datastore", "counter-datastore" ],
"uUID" : "3b6939c9-2551-4537-a8ec-291a0c5ed434"
}, {
"externalHostName" : "localhost",
"externalIP" : "xxx.xxx.xxx.xxx",
"internalHostName" : "localhost",
"internalIP" : "xxx.xxx.xxx.xxx",
"isUp" : true,
"pod" : "gateway",
"reachable" : true,
"region" : "dc-1",
"tags" : {
"property" : [ {
"name" : "jmx.rmi.port",
"value" : "1101"
}, {
"name" : "http.port",
"value" : "8998"
}, {
"name" : "http.management.port",
"value" : "8082"
}, {
"name" : "started.at",
"value" : "1458121818267"
}, {
"name" : "http.ssl.flag",
"value" : "true"
}, {
"name" : "rpc.port",
"value" : "4528"
}, {
"name" : "Profile",
"value" : "MessageProcessor"
}, {
"name" : "startup.time",
"value" : "2 minute(s) 25 second(s)"
}, {
"name" : "up.time",
"value" : "333 hour(s) 53 minute(s) 5 second(s)"
} ]
},
"type" : [ "message-processor" ],
"uUID" : "5de18053-0512-4230-9e32-add42c75bfd6"
} ]
==> 200
(...)
LinkedIn
Twitter
