1

We are using Cassandra for our services with a cluster of 4 data center and 6 nodes in each DC. Recently we update our node SKU by first adding 6 new nodes in same data center, migrating the data from existing nodes to new nodes and then removed old nodes by running either nodetool decommission or nodetool removenode command.

After this we started seeing that new Cassandra nodes are frequently going down and throwing error A node with address already exists, cancelling join while trying to come up. It then doesnt come up. I need to remove the node which is in DN status and restart Cassandra service again to make this node up. But after few days same thing happens again.

Wondering why we started seeing this issue suddenly and how can this be fixed. We are using Apache Cassandra 3.11.10 version.
Error logs : ERROR [main] 2023-04-17 17:28:29,128 CassandraDaemon.java:803 - Exception encountered during startup java.lang.RuntimeException: A node with address DM3PEPF00014567/10.115.155.147 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.     at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:632) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:880) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.StorageService.initServer(StorageService.java:738) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.StorageService.initServer(StorageService.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395) [apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633) [apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786) [apache-cassandra-3.11.10.jar:3.11.10]
This node was up and running for last few months and suddenly started throwing this error. I can confirm that node with same IP address doesnt exists in cluster.

Here is the nodetool status for this DC : Nodetool status output

When this happens I usually run nodetool removenode <host ID> command which will remove this node and then restart Cassandra on that cluster which will bring it up. But wondeirng how to fix this issue permanently as we are seeing this very intermittently.

2
  • Random question - Do you have a node or nodes that have replace_address in the cassandra.yaml file left over from the initial startup? That would have to be deleted before the node is restarted.
    – LHWizard
    Apr 28, 2023 at 15:02
  • No, I do not have replace_address in the cassandra.yaml. I also noticed that there is a solution to this in cassandra version 4.0 or later as mentioned here. wanted to check if this is a knonnw bug in version earlier that 4.0. May 1, 2023 at 23:24

2 Answers 2

1

The symptoms you described indicate to me that you have two nodes in the cluster configured with the same IP address that are trying to join the cluster. However, you didn't provide enough information so it's difficult to determine what is going.

A friendly note that you need to provide sufficient details when asking a question. The general guidance is that you (a) provide a good summary of the problem that includes software/component versions, the full error message + full stack trace; (b) describe what you've tried to fix the problem, details of investigation you've done; and (c) steps to replicate the problem.

For guidance, see Asking Good Cassandra Questions I published on the Apache Cassandra website. Additionally since you're new to DBA Stack Exchange, I would recommend taking the tour and reading the guide on how to ask questions. Cheers!

1
  • Hello @Erick I have added more details to question. Hope thats helpful. Apr 18, 2023 at 0:48
0

So I've seen this error before:

A node with address DM3PEPF00014567/10.115.155.147 already exists, cancelling join.
Use cassandra.replace_address if you want to replace this node.

If you're sure that two nodes aren't coming up with the same IP address, then it's likely that this IP address was used on one of the prior nodes, and needs to be removed from cluster gossip (system.peers table on all nodes). The easy way to do this, is to run a nodetool assassinate from any "good" node in the cluster.

nodetool assassinate 10.115.155.147

Then try to rejoin the node.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.