Hi Apigeeks,
I have created a VHOST in my "org" and environment "env" for normal HTTP and HTTPS traffic. But when I do an api call to the router with given port, I am getting a connection refused status. I even restarted router after creating VHOST.
I checked the netstat -ltnp and couldn't find any VHOST ports listening.
Checked for /opt/nginx/conf.d/ and no conf files are present for the VHOST there.
In router logs I found an something like below:
2017-06-08 10:43:14,187 pool-9-thread-5 INFO BOOTSTRAP - RuntimeConfigurationServiceImpl.internalConfigureEnvironment() : No bootstrap listeners found for deploy events, skipping application deployments 2017-06-08 10:43:14,187 pool-9-thread-6 ERROR BOOTSTRAP - RuntimeConfigurationServiceImpl.dispatchToListeners() : RuntimeConfigurationServiceImpl.dispatchToListeners : Error occurred while dispatching the requests [ConfigEvent{beanName='null', path='/organizations/org/environments/env/', action=sync, beanClass=class com.apigee.messaging.config.beans.EnvironmentBean, bean=null, itemType='null', description='SYNC Environment /organizations/org/environments/env/', xid=null}] to com.apigee.application.bootstrap.listeners.RouterManagementBootstrapListener@6831d540 com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later at com.apigee.datastore.client.astyanax.AstyanaxCassandraClient.fetchColumnValueAsByteArray(AstyanaxCassandraClient.java:1469) ~[datastore-1.0.0.jar:na] at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.readFromCassandra(CassandraRepositoryDelegate.java:662) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.readFromEnvironmentsColumnFamily(CassandraRepositoryDelegate.java:597) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.loadFile(CassandraRepositoryDelegate.java:498) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.nosql.CassandraRepository.loadFile(CassandraRepository.java:97) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.RepositoryServiceImpl.load(RepositoryServiceImpl.java:298) ~[repository-impl-1.0.0.jar:na] at com.apigee.beans.BeanUtil.load(BeanUtil.java:91) ~[beans-1.0.0.jar:na] at com.apigee.messaging.config.ConfigEvent.getBean(ConfigEvent.java:127) ~[config-model-1.0.0.jar:na]
We are on OPDK version 4.16.09.
Solved! Go to Solution.
Hi All,
Resolved this issue.
Reason:
Our infra team accidentally made a clone of one of the apigee cluster node which got registered itself in zookeeper conf. This also resulted in starting of components in the wrong sequence(this cloned nodes had analytics and central components).
Also this had impact on the analytics not being pulled in UI dashboards.
Fix:
Stopped all the components on new cloned nodes.
Stopped all other components of apigee installation in proper stopping sequence as documented in Apigee Operations Guide.
Restarted all the components in proper starting sequence.
For reference Start/Stop Order.
Hi @Mohammed Zuber,
I think the APIproxies were not deployed as per this statement
No bootstrap listeners found for deploy events, skipping application deployments
Before we investigate why that happened, I see the below statement which tells us that there are some issues with the wiring
com.apigee.datastore.DataAccessException: Error while accessing datastore;Please retry later
Looks like you are having connectivity issues and especially connectivity to Cassandra.
Let's fix that first.
You can check the wiring status by running the below APIs or you can share here with us if you are okay with that
Curl -v http://mgmthost:8080/v1/servers?pod=central®ion=dc01 Curl -v http://mgmthost:8080/v1/servers?pod=gateway®ion=dc01 Curl -v http://mgmthost:8080/v1/servers?pod=analytics®ion=dc01
Also make sure you have all the required ports open as per this http://docs.apigee.com/private-cloud/latest/installation-requirements
Thanks @Maruti Chand,
I have checked for wiring, the connections seems to be okay. All ports are open and statuses are green in servers.
We have noticed that in the system.log of a working and non-working roters there is this difference: also, we checked the ports and they are all open from all R to all CS.: zk 2181, and cs 9042 9160
Is there any way to establish why com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException happens?
working Router: 2017-06-08 12:45:55,710 main INFO DATASTORE.CASSANDRA - ConnectionPoolManager.createConnectionPoolKey() : created co nnection pool key apprepo#<CS_IP>_<CS_IP>_<CS_IP> for config name application-datastore 2017-06-08 12:45:55,866 main INFO c.n.a.c.i.ConnectionPoolMBeanManager - ConnectionPoolMBeanManager.registerMonitor( ) : Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=application-datastore,ServiceType=connectionp ool 2017-06-08 12:45:55,875 main INFO c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde d() : AddHost: <CS_IP> 2017-06-08 12:45:55,876 main INFO c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde d() : AddHost: <CS_IP> 2017-06-08 12:45:55,876 main INFO c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde d() : AddHost: <CS_IP> ---> 2017-06-08 12:45:56,233 main INFO BOOTSTRAP - RuntimeConfigurationServiceImpl$1.run() : RuntimeConfigurationServiceI mpl.start : Server is registered as [router] Will proceed to sync up the configurations 2017-06-08 12:45:56,233 main INFO BOOTSTRAP - RuntimeConfigurationServiceImpl.initiateSync() : RuntimeConfigurationS erviceImpl.initiateSync : async : true 2017-06-08 12:45:56,236 main INFO BOOTSTRAP - RuntimeConfigurationServiceImpl.internalInitiateSync() : RuntimeConfig urationServiceImpl.internalInitiateSync : Organizations : [org, VALIDATE] non-working Router: 2017-06-08 12:00:36,221 main INFO DATASTORE.CASSANDRA - ConnectionPoolManager.createConnectionPoolKey() : created co nnection pool key apprepo#<CS_IP>_<CS_IP>_<CS_IP> for config name application-datastore 2017-06-08 12:00:36,420 main INFO c.n.a.c.i.ConnectionPoolMBeanManager - ConnectionPoolMBeanManager.registerMonitor( ) : Registering mbean: com.netflix.MonitoredResources:type=ASTYANAX,name=application-datastore,ServiceType=connectionp ool 2017-06-08 12:00:36,434 main INFO c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde d() : AddHost: <CS_IP> 2017-06-08 12:00:36,435 main INFO c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde d() : AddHost: <CS_IP> 2017-06-08 12:00:36,435 main INFO c.n.a.c.i.CountingConnectionPoolMonitor - CountingConnectionPoolMonitor.onHostAdde d() : AddHost: <CS_IP> ----> 2017-06-08 12:00:48,511 main ERROR S.REPOSITORY.CASSANDRA - ApplicationRepoServerListener.initializeKeySpace() : Conn ectionException while creating column family in apprepo keyspace for cluster:[application-datastore] com.netflix.astyanax.connectionpool.exceptions.PoolTimeoutException: PoolTimeoutException: [host=None(<CS_IP>):0, late ncy=6001(6001), attempts=3]Timed out waiting for connection at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.waitForConnection(SimpleHostConnectionPoo l.java:231) ~[astyanax-core-1.56.43.jar:na] at com.netflix.astyanax.connectionpool.impl.SimpleHostConnectionPool.borrowConnection(SimpleHostConnectionPool .java:198) ~[astyanax-core-1.56.43.jar:na] at com.netflix.astyanax.connectionpool.impl.LeastOutstandingExecuteWithFailover.borrowConnection(LeastOutstand ingExecuteWithFailover.java:74) ~[astyanax-core-1.56.43.jar:na] at com.netflix.astyanax.connectionpool.impl.AbstractExecuteWithFailoverImpl.tryOperation(AbstractExecuteWithFa iloverImpl.java:117) ~[astyanax-core-1.56.43.jar:na] at com.netflix.astyanax.connectionpool.impl.AbstractHostPartitionConnectionPool.executeWithFailover(AbstractHo stPartitionConnectionPool.java:338) ~[astyanax-core-1.56.43.jar:na] at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.executeOperation(ThriftKeyspaceImpl.java:493) ~[astyanax-thr ift-1.56.43.jar:na] at com.netflix.astyanax.thrift.ThriftKeyspaceImpl.describeKeyspace(ThriftKeyspaceImpl.java:244) ~[astyanax-thr ift-1.56.43.jar:na] at com.apigee.services.repository.nosql.ApplicationRepoServerListener.initializeKeySpace(ApplicationRepoServer Listener.java:153) ~[repository-impl-1.0.0.jar:na] at com.apigee.services.repository.nosql.CassandraRepositoryDelegate.start(CassandraRepositoryDelegate.java:180 ) [repository-impl-1.0.0.jar:na] at com.apigee.services.repository.nosql.CassandraRepository.start(CassandraRepository.java:33) [repository-imp l-1.0.0.jar:na] <br>
You mentioned all the servers are in green but can you check if your cassandra nodes are listening on 9160?
Do you have any properties defined in /opt/apigee/customer/application/cassandra.properties?
Yes all cassandra nodes are listening on port 9160 and we don't have any custom properties defined in /opt/apigee/customer/application/cassandra.properties
Hi All,
Resolved this issue.
Reason:
Our infra team accidentally made a clone of one of the apigee cluster node which got registered itself in zookeeper conf. This also resulted in starting of components in the wrong sequence(this cloned nodes had analytics and central components).
Also this had impact on the analytics not being pulled in UI dashboards.
Fix:
Stopped all the components on new cloned nodes.
Stopped all other components of apigee installation in proper stopping sequence as documented in Apigee Operations Guide.
Restarted all the components in proper starting sequence.
For reference Start/Stop Order.