Apigee Hybrid Cassandra Data Replication Issue

[kyisoethinkokyi]

I set up 2 regions apigee-hybrid deployment recently. I have completed all the steps for multi region deployment. But for cassandra data replication, when I run this command

"kubectl -n apigee get apigeeds -o json | jq .items[].status.cassandraDataReplication"

I got, 

{}

I don't get any response shown in the documentation. But  when I run status command,

kubectl -n apigee exec -it apigee-cassandra-default-0 -- nodetool -u jmxuser -pw password status

I got following response:

Datacenter: dc-1
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  x.x.x.x   1.52 MiB   256          0.0%              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ra-1
UN  x.x.x.x   1.62 MiB   256          0.0%              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ra-1
UN  x.x.x.x  1.74 MiB   256          0.0%              xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ra-1
Datacenter: dc-2
================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns (effective)  Host ID                               Rack
UN  x.x.x.x  1.87 MiB   256          100.0%            xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ra-1
UN  x.x.x.x  1.8 MiB    256          100.0%            xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ra-1
UN  x.x.x.x  1.56 MiB   256          100.0%            xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx  ra-1

Is it okay? Or do I have an issue?

[azharali49]

Does the output of below commands shows all the 6 cassandra PODS?

kubectl -n apigee exec -it apigee-cassandra-default-0 -- nodetool -u jmxuser -pw password info

kubectl -n apigee exec -it apigee-cassandra-default-0 -- nodetool -u jmxuser -pw password describecluster

kubectl -n apigee exec -it apigee-cassandra-default-0 -- nodetool -u jmxuser -pw password status

kubectl -n apigee exec -it apigee-cassandra-default-0 -- nodetool -u jmxuser -pw password gossipinfo

You can also refer the article Cassandra data replication failure

[shayan]

Hello @azharali49 @kyisoethinkokyi ,

What is the solution for this problem.
I got response from all these commands.

[ssvaidyanathan]

@shayan - can you open a new post with your issue and your findings so that we can help resolve the problem you are facing?

[ayos]

@shayan  Could you please include the  hybrid release version which you are installing as well as state of the cassandra pods ( any status or events messages) from the kubectl describe of the cassandra pods

[shayan]

Hello @ayos @ssvaidyanathan,

The issue lies in the Apigee documentation—while the multi-region setup documentation only specifies ports 7001 and 7199, enabling ports 8778 and 54110 as well allowed the replication to function correctly.