adding cassandra node in new data center

[Kassav]

Hello,

I'm adding new datacenter to a multiple datacenter infrastructure.

Now i'm adding 3 cassandra nodes using steps in the documentation.

I'm encountering this error on 2 new nodes:

WARN [GossipStage:1] 2024-07-08 11:04:02,205 TokenMetadata.java:196 - Token 56713727xxxxxxxxx42 changing ownership from /x.x.x.x to /y.y.y.y

And after this an existing node was removed (which is not what was expected) an the new host is added with it's new datacenter

The third host, i'm getting this error; 

INFO [main] 2024-07-09 11:33:04,896 StorageService.java:1138 - JOINING: getting bootstrap token
ERROR [main] 2024-07-09 11:33:04,901 CassandraDaemon.java:399 - Fatal configuration error
org.apache.cassandra.exceptions.ConfigurationException: Bootstrapping to existing token 11342xxxxxx256070685 is not allowed (decommission/removenode the old node first).

Any recommendations to add new cassandra nodes in a new datacenter ? did i missed a step ?

Thank you

[Hilda_Arteaga]

Hello @Kassav 

Are you still having issues with these? Can you see your nodes now?

It was recently discovered in a case that the Cassandra node may take an unexpectedly long time to join the cluster and replicate the data from the nodes. In that case, the node may be joining the cluster for hours/days;if/at the same time, another node(s) goes down, which may result in Cassandra outage if the cluster has less than 2 nodes up and running. Such an outage will also cause the common-mp-fault-rate alerts from the impacted region. If this is the case, you can have a look at this OMG and use it for reference (omg/73484). 

To check whether a similar problem is related you can:

1. Check Cassandra's combined metrics for the problematic pod. Here's what the diamond01 pod experienced during the OMG mentioned before.
2. SSH to one of the cluster's nodes and use nodetool status to check whether the number of up-and-running nodes in the Cassandra cluster is less than 2 in that region.

If, after checking, you think that your alerts are related to this issue, please escalate to the Apigee Dataservices team.

Apologies for the delay in our response and thanks for sharing your inquiries in this forum. We hope this information is helpful.

[Kassav · Accepted solution]

Hello @Hilda_Arteaga 

There were collision in cassandra initial token, when adding a new dc which value is not greater than other dc numbers

Solution was to update conf_cassandra_initial_token to custon unique values

[Hilda_Arteaga]

Hi @Kassav thansk for shairng this