This website uses Cookies. Click Accept to agree to our website's cookie use as described in our Privacy Policy. Click Preferences to customize your cookie settings.
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for
- Google Cloud
- Cloud Forums
- Apigee
- Re: Stuck with error in edge router - Checking for...
Topic Options
- Subscribe to RSS Feed
- Mark Topic as New
- Mark Topic as Read
- Float this Topic for Current User
- Bookmark
- Subscribe
- Mute
- Printer Friendly Page
Solved
LinkedIn
Twitter
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Trying the 9 node setup. node1 to node3 setup is done( edge management server), cassandra, zookeeper.
trying this setup on node4. stuck here. Your help is really appreciated.
ENVIRONMENT:
OS: RHEL
Apigee Version: 4.51.00
-----
[root@rhel-apigee9-set2-vm-04 apigee]# java -version
openjdk version "1.8.0_312"
OpenJDK Runtime Environment (build 1.8.0_312-b07)
OpenJDK 64-Bit Server VM (build 25.312-b07, mixed mode)
---
[root@rhel-apigee9-set2-vm-04 apigee]# uname -a
Linux ydx-stage-msa-rhel-apigee9-set2-vm-04 4.18.0-305.17.1.el8_4.x86_64 #1 SMP Mon Aug 30 07:26:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux
-----------
/opt/apigee/apigee-setup/bin/setup.sh -f /tmp/apigee/response-edge.txt -p rmp
------------
apigee-service: disable confirmation prompts
0 * * * * nice -n 19 ionice -c3 /usr/sbin/logrotate -f /opt/apigee/edge-router/logrotate/logrotate.conf
removing logrotate cron entry
adding logrotate cron entry
logrotate reconfigure for edge-router
updated property conf_datastore_push_cron_duration in file /opt/apigee/token/application/router.properties
apigee-configutil: edge-router: # OK
apigee-service: edge-router: edge-router is not running
apigee-configutil: edge-router: # OK
apigee-service: edge-router: Not running (NO_LOCKFILE)
apigee-service: edge-router: status=2, continuing
apigee-service: edge-router: OK
apigee-service: edge-router: edge-router is running
Checking for router on 127.0.0.1 port 8081 .........................
============= port currently running
[root@rhel-apigee9-set2-vm-04 apigee]# netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:5355 0.0.0.0:* LISTEN 890/systemd-resolve
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1647/sshd
tcp 0 0 127.0.0.1:9080 0.0.0.0:* LISTEN 81976/nginx: master
tcp6 0 0 :::5355 :::* LISTEN 890/systemd-resolve
tcp6 0 0 :::22 :::* LISTEN 1647/sshd
-----------------------------------
[root@apigee9-set2-vm-04 sbin]# getenforce
Permissive
--------
[root@apigee9-set2-vm-04 sbin]# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
#::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
-----------------
this is what the healthcheck of the
apigee-services on all the nodes:
host01 | CHANGED | rc=0 >>
+ apigee-service apigee-cassandra status
apigee-service: apigee-cassandra: OK
+ apigee-service apigee-openldap status
apigee-service: apigee-openldap: OK
+ apigee-service apigee-zookeeper status
apigee-service: apigee-zookeeper: OK
+ apigee-service edge-management-server status
apigee-service: edge-management-server: OK
+ apigee-service edge-ui status
apigee-service: edge-ui: OK
host02 | CHANGED | rc=0 >>
+ apigee-service apigee-cassandra status
apigee-service: apigee-cassandra: OK
+ apigee-service apigee-zookeeper status
apigee-service: apigee-zookeeper: OK
host03 | CHANGED | rc=0 >>
+ apigee-service apigee-cassandra status
apigee-service: apigee-cassandra: OK
+ apigee-service apigee-zookeeper status
apigee-service: apigee-zookeeper: OK
host04 | FAILED | rc=1 >>
+ apigee-service edge-router status
apigee-service: edge-router: Not running (DEAD)
apigee-all: Error: status failed on [edge-router]non-zero return code
host05 | FAILED | rc=1 >>
+ apigee-service edge-router status
apigee-service: edge-router: Not running (DEAD)
apigee-all: Error: status failed on [edge-router]non-zero return code
--------
Edge router error is:
2021-11-24 18:05:51,559 main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Starting
2021-11-24 18:06:01,611 main INFO o.a.c.f.i.CuratorFrameworkImpl - CuratorFrameworkImpl.start() : Default schema
2021-11-24 18:06:01,612 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Successfully initialized the service : ZooKeeperService
2021-11-24 18:06:01,612 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : MetricsService has these dependencies [ [NameRegistration{name=CommunicationService}] ]
2021-11-24 18:06:01,613 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : OrganizationService has these dependencies [ [NameRegistration{name=CommunicationService}, NameRegistration{name=RepositoryService}] ]
2021-11-24 18:06:01,613 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : RepositoryService has these dependencies [ [NameRegistration{name=ZooKeeperService}, NameRegistration{name=DataStoreService}, NameRegistration{name=ServerMetricsService}] ]
2021-11-24 18:06:01,614 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Initializing service : RepositoryService that implements InitializeAwareService
2021-11-24 18:06:01,657 main INFO SERVICES.REPOSITORY - RepositoryServiceImpl.initRepository() : repository name=apprepo is using class=com.apigee.services.repository.nosql.CassandraRepository
2021-11-24 18:06:01,659 main INFO SERVICES.REPOSITORY - RepositoryServiceImpl.initRepository() : repository name=zookeeper is using class=com.apigee.services.repository.zookeeper.ZKRepository
2021-11-24 18:06:01,666 main INFO SERVICES.REPOSITORY - RepositoryServiceImpl.initRepository() : repository name=localrepo is using class=com.apigee.services.repository.filesystem.FileSystemRepository
2021-11-24 18:06:01,685 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Successfully initialized the service : RepositoryService
2021-11-24 18:06:01,686 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.getUndeployedRequirements() : ServerRegistrationService has these dependencies [ [NameRegistration{name=RepositoryService}, NameRegistration{name=CommunicationService}, NameRegistration{name=ComponentConfigurationService}, NameRegistration{name=ServerMetricsService}] ]
2021-11-24 18:06:01,687 main INFO KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Initializing service : ServerRegistrationService that implements InitializeAwareService
2021-11-24 18:06:21,635 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed
2021-11-24 18:06:21,746 main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: SUSPENDED
2021-11-24 18:06:21,757 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:06:24,746 Curator-ConnectionStateManager-0 WARN o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.checkSessionExpiration() : Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 3000. Adjusted session timeout ms: 3000
2021-11-24 18:06:24,749 main-EventThread WARN o.a.c.ConnectionState - ConnectionState.handleExpiredSession() : Session expired event received
2021-11-24 18:06:27,747 Curator-ConnectionStateManager-0 WARN o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.checkSessionExpiration() : Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 3001. Adjusted session timeout ms: 3000
2021-11-24 18:06:41,868 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:06:41,871 Curator-ConnectionStateManager-0 WARN o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.checkSessionExpiration() : Session timeout has elapsed while SUSPENDED. Injecting a session expiration. Elapsed ms: 14124. Adjusted session timeout ms: 3000
2021-11-24 18:06:41,871 main-EventThread INFO o.a.c.f.s.ConnectionStateManager - ConnectionStateManager.postState() : State change: LOST
2021-11-24 18:06:41,872 main-EventThread WARN o.a.c.ConnectionState - ConnectionState.handleExpiredSession() : Session expired event received
2021-11-24 18:07:22,002 main-EventThread ERROR o.a.c.ConnectionState - ConnectionState.checkState() : Authentication failed
2021-11-24 18:07:22,104 main INFO ZOOKEEPER - ZooKeeperServiceImpl.exists() : Retry path existence path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:07:22,104 main ERROR ZOOKEEPER - ZooKeeperServiceImpl.exists() : Could not detect existence of path: /types, reason: KeeperErrorCode = ConnectionLoss for /types
2021-11-24 18:07:22,115 main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Got a life cycle exception while initializing service [ServerRegistrationService, Error while checking path existence for path : /types] : {}
com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /types
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:407)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:384)
at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.apigee.server.metrics.aop.MetricsEmissionInvocationHandler.invoke(MetricsEmissionInvocationHandler.java:79)
at com.sun.proxy.$Proxy5.exists(Unknown Source)
at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:203)
at com.apigee.registration.ServerRegistrationServiceImpl.getServerTypes(ServerRegistrationServiceImpl.java:377)
at com.apigee.registration.ServerRegistrationServiceImpl.initResolutionScopes(ServerRegistrationServiceImpl.java:399)
at com.apigee.registration.ServerRegistrationServiceImpl.init(ServerRegistrationServiceImpl.java:201)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:165)
at com.apigee.kernel.service.deployment.ServiceDeployer.initDependantServices(ServiceDeployer.java:481)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:184)
at com.apigee.kernel.MicroKernel.initAll(MicroKernel.java:283)
at com.apigee.kernel.MicroKernel.init(MicroKernel.java:132)
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:138)
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /types
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:399)
... 19 common frames omitted
2021-11-24 18:07:22,116 main ERROR KERNEL - MicroKernel.initAll() : Error in Initializing the deployment : RepositoryService. Exception: {}
com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /types
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:407)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:384)
at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.apigee.server.metrics.aop.MetricsEmissionInvocationHandler.invoke(MetricsEmissionInvocationHandler.java:79)
at com.sun.proxy.$Proxy5.exists(Unknown Source)
at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:203)
at com.apigee.registration.ServerRegistrationServiceImpl.getServerTypes(ServerRegistrationServiceImpl.java:377)
at com.apigee.registration.ServerRegistrationServiceImpl.initResolutionScopes(ServerRegistrationServiceImpl.java:399)
at com.apigee.registration.ServerRegistrationServiceImpl.init(ServerRegistrationServiceImpl.java:201)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:165)
at com.apigee.kernel.service.deployment.ServiceDeployer.initDependantServices(ServiceDeployer.java:481)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:184)
at com.apigee.kernel.MicroKernel.initAll(MicroKernel.java:283)
at com.apigee.kernel.MicroKernel.init(MicroKernel.java:132)
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:138)
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /types
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:399)
... 19 common frames omitted
2021-11-24 18:07:22,117 main ERROR KERNEL - MicroKernel.main() : Exception in starting up the kernel:
com.apigee.zookeeper.ZooKeeperException: Error while checking path existence for path : /types
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:407)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:384)
at com.apigee.services.repository.zookeeper.ZKRepository.exists(ZKRepository.java:254)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at com.apigee.server.metrics.aop.MetricsEmissionInvocationHandler.invoke(MetricsEmissionInvocationHandler.java:79)
at com.sun.proxy.$Proxy5.exists(Unknown Source)
at com.apigee.services.repository.RepositoryServiceImpl.exists(RepositoryServiceImpl.java:203)
at com.apigee.registration.ServerRegistrationServiceImpl.getServerTypes(ServerRegistrationServiceImpl.java:377)
at com.apigee.registration.ServerRegistrationServiceImpl.initResolutionScopes(ServerRegistrationServiceImpl.java:399)
at com.apigee.registration.ServerRegistrationServiceImpl.init(ServerRegistrationServiceImpl.java:201)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:165)
at com.apigee.kernel.service.deployment.ServiceDeployer.initDependantServices(ServiceDeployer.java:481)
at com.apigee.kernel.service.deployment.ServiceDeployer.init(ServiceDeployer.java:184)
at com.apigee.kernel.MicroKernel.initAll(MicroKernel.java:283)
at com.apigee.kernel.MicroKernel.init(MicroKernel.java:132)
at com.apigee.kernel.MicroKernel.start(MicroKernel.java:138)
at com.apigee.kernel.MicroKernel.main(MicroKernel.java:90)
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /types
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1111)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1139)
at com.apigee.zookeeper.impl.ZooKeeperServiceImpl.exists(ZooKeeperServiceImpl.java:399)
... 19 common frames omitted
----------------
0
9
1,825
Topic Labels
- Labels:
-
Private Cloud Deployment
9 REPLIES 9
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
2021-11-24 18:07:22,115 main ERROR KERNEL.DEPLOYMENT - ServiceDeployer.init() : ServiceDeployer.init() : Got a life cycle exception while initializing service [ServerRegistrationService, Error while checking path existence for path : /types] : {}
Your edge-routers are failing to connect to Zookeeper. You should check connectivity (routing, firewall rules) from your edge-router nodes to your Zookeeper nodes.
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks @cjking
I have done these things already.
sudo systemctl disable firewalld; sudo systemctl stop firewalld
In the cloud also, these are part of same subnet and all vms have private ip & only public ip associated to node01 alone. These is NO network barrier to connect between these nodes.
This is on RHEL 8.0
Any other suggestions please apart from below checks?
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 apigee]# iptables -L -n
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
---------
I am able to do telnet to all the zookeeper nodes from my node4 & node5.
---
[root@rhel-apigee9-set2-vm-04 apigee]# echo| telnet 172.168.1.197 2181
Trying 172.168.1.197...
Connected to 172.168.1.197.
Escape character is '^]'.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-04 apigee]# echo| telnet 172.168.1.15 2181
Trying 172.168.1.15...
Connected to 172.168.1.15.
Escape character is '^]'.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-04 apigee]# echo| telnet 172.168.1.142 2181
Trying 172.168.1.142...
Connected to 172.168.1.142.
Escape character is '^]'.
Connection closed by foreign host.
[
[root@rhel-apigee9-set2-vm-05 apigee]# echo| telnet 172.168.1.197 2181
Trying 172.168.1.197...
Connected to 172.168.1.197.
Escape character is '^]'.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-05 apigee]# echo| telnet 172.168.1.15 2181
Trying 172.168.1.15...
Connected to 172.168.1.15.
Escape character is '^]'.
Connection closed by foreign host.
[root@rhel-apigee9-set2-vm-05 apigee]# echo| telnet 172.168.1.142 2181
Trying 172.168.1.142...
Connected to 172.168.1.142.
Escape character is '^]'.
Connection closed by foreign host.
[
--
Hostlist:
host01 ansible_host=172.168.1.197
host02 ansible_host=172.168.1.15
host03 ansible_host=172.168.1.142
host04 ansible_host=172.168.1.193
host05 ansible_host=172.168.1.192
host06 ansible_host=172.168.1.195
host07 ansible_host=172.168.1.194
host08 ansible_host=172.168.1.191
host09 ansible_host=172.168.1.149
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I'd suggest giving the ruok and stat commands from https://docs.apigee.com/api-platform/troubleshoot/diagnostic-logs/zookeeper-problems?hl=en a try, both locally on your ZK nodes and remotely using nc from your router nodes. It's possible your ZK cluster isn't healthy.
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have run it on all the vms as suggested.
[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee]# echo "ruok" | nc localhost 2181
imok
[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee ]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 15883
Sent: 15883
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-01 apigee]# echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 15886
zk_packets_sent 15886
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-01 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:41614[0](queued=0,recved=1,sent=0)
/172.168.1.197:44810[1](queued=0,recved=2238,sent=2238)
/172.168.1.197:44610[1](queued=0,recved=13636,sent=13637)
Latency min/avg/max: 0/0/22
Received: 15890
Sent: 15890
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-01 apigee]# echo cons | nc localhost 2181
/172.168.1.197:44810[1](queued=0,recved=2241,sent=2241,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=47969889,llat=0,minlat=0,avglat=0,maxlat=5)
/0:0:0:0:0:0:0:1:41656[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13657,sent=13658,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2d8a,lzxid=0xffffffffffffffff,lresp=47974178,llat=0,minlat=0,avglat=0,maxlat=22)
----------------------------------------------------------------------------------
[root@rhel-apigee9-set2-vm-02 apigee]# echo ruok | nc localhost 2181
imok
[root@rhel-apigee9-set2-vm-02 apigee]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-02 apigee]# echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 3
zk_packets_sent 2
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-02 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:37784[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-02 apigee]# echo cons | nc localhost 2181
/0:0:0:0:0:0:0:1:37786[0](queued=0,recved=1,sent=0)
----------------------------------------------------------------------------------
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo ruok | nc localhost 2181
imok
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 3
zk_packets_sent 2
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:37480[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo cons | nc localhost 2181
/0:0:0:0:0:0:0:1:37482[0](queued=0,recved=1,sent=0)
[apigee@rhel-apigee9-set2-vm-03 ~]$
----------------------------------------------------------------------------------
FROM NODE4:(edge-router machine)
--------------------------------
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.197 2181
imok[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16224
Sent: 16224
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.197 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 16225
zk_packets_sent 16225
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.197:44810[1](queued=0,recved=2290,sent=2290)
/172.168.1.193:36732[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13910,sent=13911)
Latency min/avg/max: 0/0/22
Received: 16226
Sent: 16226
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.197 2181
/172.168.1.197:44810[1](queued=0,recved=2290,sent=2290,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=48623605,llat=0,minlat=0,avglat=0,maxlat=5)
/172.168.1.193:36734[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13910,sent=13911,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2e5b,lzxid=0xffffffffffffffff,lresp=48634177,llat=1,minlat=0,avglat=0,maxlat=22)
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.15 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 8
zk_packets_sent 7
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.193:57486[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.15 2181
/172.168.1.193:57488[0](queued=0,recved=1,sent=0)
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.142 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 8
zk_packets_sent 7
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.193:42672[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.142 2181
/172.168.1.193:42674[0](queued=0,recved=1,sent=0)
------------------
From Node5(edge-router):
========================
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.197 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16198
Sent: 16198
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.197 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 16199
zk_packets_sent 16199
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:59548[0](queued=0,recved=1,sent=0)
/172.168.1.197:44810[1](queued=0,recved=2289,sent=2289)
/172.168.1.197:44610[1](queued=0,recved=13890,sent=13891)
Latency min/avg/max: 0/0/22
Received: 16200
Sent: 16200
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.197 2181
/172.168.1.197:44810[1](queued=0,recved=2289,sent=2289,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=48610261,llat=0,minlat=0,avglat=0,maxlat=5)
/172.168.1.192:59550[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13890,sent=13891,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2e48,lzxid=0xffffffffffffffff,lresp=48614211,llat=0,minlat=0,avglat=0,maxlat=22)
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.15 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 13
zk_packets_sent 12
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:50260[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.15 2181
/172.168.1.192:50262[0](queued=0,recved=1,sent=0)
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.142 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 13
zk_packets_sent 12
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:58846[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.142 2181
/172.168.1.192:58848[0](queued=0,recved=1,sent=0)
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Hi @cjking
Thanks for helping me to narrow down to issue. Please find the below diagnostic information on same. please look and suggest anything to be done here.
=================================
[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee]# echo "ruok" | nc localhost 2181
imok
[root@ydx-stage-msa-rhel-apigee9-set2-vm-01 apigee]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 15883
Sent: 15883
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-01 apigee]# echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 15886
zk_packets_sent 15886
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-01 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:41614[0](queued=0,recved=1,sent=0)
/172.168.1.197:44810[1](queued=0,recved=2238,sent=2238)
/172.168.1.197:44610[1](queued=0,recved=13636,sent=13637)
Latency min/avg/max: 0/0/22
Received: 15890
Sent: 15890
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-01 apigee]# echo cons | nc localhost 2181
/172.168.1.197:44810[1](queued=0,recved=2241,sent=2241,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=47969889,llat=0,minlat=0,avglat=0,maxlat=5)
/0:0:0:0:0:0:0:1:41656[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13657,sent=13658,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2d8a,lzxid=0xffffffffffffffff,lresp=47974178,llat=0,minlat=0,avglat=0,maxlat=22)
----------------------------------------------------------------------------------
[root@rhel-apigee9-set2-vm-02 apigee]# echo ruok | nc localhost 2181
imok
[root@rhel-apigee9-set2-vm-02 apigee]# echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-02 apigee]# echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 3
zk_packets_sent 2
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-02 apigee]# echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:37784[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-02 apigee]# echo cons | nc localhost 2181
/0:0:0:0:0:0:0:1:37786[0](queued=0,recved=1,sent=0)
----------------------------------------------------------------------------------
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo ruok | nc localhost 2181
imok
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo srvr | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 2
Sent: 1
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo mntr | nc localhost 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 3
zk_packets_sent 2
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo stat | nc localhost 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/0:0:0:0:0:0:0:1:37480[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 4
Sent: 3
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[apigee@rhel-apigee9-set2-vm-03 ~]$ echo cons | nc localhost 2181
/0:0:0:0:0:0:0:1:37482[0](queued=0,recved=1,sent=0)
[apigee@rhel-apigee9-set2-vm-03 ~]$
----------------------------------------------------------------------------------
FROM NODE4:(edge-router machine)
--------------------------------
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.197 2181
imok[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16224
Sent: 16224
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@ydx-stage-msa-rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.197 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 16225
zk_packets_sent 16225
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.197:44810[1](queued=0,recved=2290,sent=2290)
/172.168.1.193:36732[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13910,sent=13911)
Latency min/avg/max: 0/0/22
Received: 16226
Sent: 16226
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.197 2181
/172.168.1.197:44810[1](queued=0,recved=2290,sent=2290,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=48623605,llat=0,minlat=0,avglat=0,maxlat=5)
/172.168.1.193:36734[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13910,sent=13911,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2e5b,lzxid=0xffffffffffffffff,lresp=48634177,llat=1,minlat=0,avglat=0,maxlat=22)
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.15 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 8
zk_packets_sent 7
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.193:57486[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.15 2181
/172.168.1.193:57488[0](queued=0,recved=1,sent=0)
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]#
[root@rhel-apigee9-set2-vm-04 nginx]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-04 nginx]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 7
Sent: 6
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo mntr | nc 172.168.1.142 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 8
zk_packets_sent 7
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[root@rhel-apigee9-set2-vm-04 nginx]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.193:42672[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 9
Sent: 8
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-04 nginx]# echo cons | nc 172.168.1.142 2181
/172.168.1.193:42674[0](queued=0,recved=1,sent=0)
------------------
From Node5(edge-router):
========================
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.197 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/22
Received: 16198
Sent: 16198
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.197 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 22
zk_min_latency 0
zk_packets_received 16199
zk_packets_sent 16199
zk_num_alive_connections 3
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 36
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.197 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:59548[0](queued=0,recved=1,sent=0)
/172.168.1.197:44810[1](queued=0,recved=2289,sent=2289)
/172.168.1.197:44610[1](queued=0,recved=13890,sent=13891)
Latency min/avg/max: 0/0/22
Received: 16200
Sent: 16200
Connections: 3
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.197 2181
/172.168.1.197:44810[1](queued=0,recved=2289,sent=2289,sid=0x1000110f6320003,lop=PING,est=1637778625347,to=40000,lcxid=0x0,lzxid=0xffffffffffffffff,lresp=48610261,llat=0,minlat=0,avglat=0,maxlat=5)
/172.168.1.192:59550[0](queued=0,recved=1,sent=0)
/172.168.1.197:44610[1](queued=0,recved=13890,sent=13891,sid=0x1000110f6320001,lop=PING,est=1637778539802,to=40000,lcxid=0x2e48,lzxid=0xffffffffffffffff,lresp=48614211,llat=0,minlat=0,avglat=0,maxlat=22)
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.15 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.15 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 13
zk_packets_sent 12
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state follower
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 34
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.15 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:50260[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: follower
Node count: 208
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.15 2181
/172.168.1.192:50262[0](queued=0,recved=1,sent=0)
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]#
[root@rhel-apigee9-set2-vm-05 apigee]# echo ruok | nc 172.168.1.142 2181
imok[root@rhel-apigee9-set2-vm-05 apigee]# echo srvr | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Latency min/avg/max: 0/0/0
Received: 12
Sent: 11
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo mntr | nc 172.168.1.142 2181
zk_version 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
zk_avg_latency 0
zk_max_latency 0
zk_min_latency 0
zk_packets_received 13
zk_packets_sent 12
zk_num_alive_connections 1
zk_outstanding_requests 0
zk_server_state leader
zk_znode_count 208
zk_watch_count 0
zk_ephemerals_count 0
zk_approximate_data_size 16188
zk_open_file_descriptor_count 36
zk_max_file_descriptor_count 262144
zk_fsync_threshold_exceed_count 0
zk_followers 2
zk_synced_followers 2
zk_pending_syncs 0
zk_last_proposal_size 36
zk_max_proposal_size 1136
zk_min_proposal_size 32
[root@rhel-apigee9-set2-vm-05 apigee]# echo stat | nc 172.168.1.142 2181
Zookeeper version: 3.4.14-4c25d480e66aadd371de8bd2fd8da255ac140bcf, built on 03/06/2019 16:18 GMT
Clients:
/172.168.1.192:58846[0](queued=0,recved=1,sent=0)
Latency min/avg/max: 0/0/0
Received: 14
Sent: 13
Connections: 1
Outstanding: 0
Zxid: 0x20000002e
Mode: leader
Node count: 208
Proposal sizes last/min/max: 36/32/1136
[root@rhel-apigee9-set2-vm-05 apigee]# echo cons | nc 172.168.1.142 2181
/172.168.1.192:58848[0](queued=0,recved=1,sent=0)
================================
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I have also created a case 1508350
Following up with support team on this. Even I have updated this thread, but didn't see my update.
I have also followed the procedure you have given and attached all the diagnostic logs in the case.
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Error while checking path existence for path : /types] : {}
This is pretty 100% problem connecting to ZK. Are you using Apigee mTLS?
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
I haven't used any mTLS for this setup. its failing during the installation itself.
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Reply posted on
--/--/---- --:-- AM
Post Options
- Mark as New
- Bookmark
- Subscribe
- Mute
- Subscribe to RSS Feed
- Permalink
- Report Inappropriate Content
Thanks for your time and help in answering my question @Denis_KALITVI @cjking
This is the root cause of the issue:
I had to make below settings on all the apigee nodes for the installation wait for the port to be available and installation to be succeeded.
echo "PORT_TIMEOUT=300" > /opt/apigee/customer/defaults.sh; chown apigee:apigee /opt/apigee/customer/defaults.sh
Once you run the bootstrap and install command, update below timeout settings on respective service nodes according to your installation topology you setup, before deploying any apigee service. This will solve the issue.
This can be reproduced at RHEL setup(using RHEL 8.0 from azure cloud marketplace vm) . I didn't face this issue in CentOS. I think some fix is needed in the Product installation script.
echo "conf_zookeeper_session.timeout=300000" > /opt/apigee/customer/application/management-server.properties; chown apigee:apigee /opt/apigee/customer/application/management-server.properties
echo "conf_zookeeper_session.timeout=300000" > /opt/apigee/customer/application/message-processor.properties;chown apigee:apigee /opt/apigee/customer/application/message-processor.properties
echo "conf_zookeeper_session.timeout=300000" > /opt/apigee/customer/application/router.properties; chown apigee:apigee /opt/apigee/customer/application/router.properties
echo "conf_zookeeper_session.timeout=300000" > /opt/apigee/customer/application/qpid-server.properties; chown apigee:apigee /opt/apigee/customer/application/qpid-server.properties
echo "conf_zookeeper_session.timeout=300000" > /opt/apigee/customer/application/postgres-server.properties; chown apigee:apigee /opt/apigee/customer/application/postgres-server.properties
echo "conf_zookeeper_session.timeout=300000" > /opt/apigee/customer/application/postgresql.properties; chown apigee:apigee /opt/apigee/customer/application/postgresql.properties
Top Labels in this Space
-
Analytics
497 -
API Hub
75 -
API Runtime
11,664 -
API Security
176 -
Apigee General
3,037 -
Apigee X
1,282 -
Developer Portal
1,909 -
Drupal Portal
43 -
Hybrid
463 -
Integrated Developer Portal
89 -
Integration
309 -
PAYG
13 -
Private Cloud Deployment
1,068 -
User Interface
76
Top Solution Authors
| User | Count |
|---|---|
| 1 | |
| 1 | |
| 1 | |
| 1 | |
| 1 |
