Cassandra nodes going down frequently and throwing error A node with address <IP Address> already exists, cancelling join

Question

We are using Cassandra for our services with a cluster of 4 data center and 6 nodes in each DC. Recently we update our node SKU by first adding 6 new nodes in same data center, migrating the data from existing nodes to new nodes and then removed old nodes by running either nodetool decommission or nodetool removenode command.

After this we started seeing that new Cassandra nodes are frequently going down and throwing error A node with address already exists, cancelling join while trying to come up. It then doesnt come up. I need to remove the node which is in DN status and restart Cassandra service again to make this node up. But after few days same thing happens again.

Wondering why we started seeing this issue suddenly and how can this be fixed. We are using Apache Cassandra 3.11.10 version.
Error logs : ERROR [main] 2023-04-17 17:28:29,128 CassandraDaemon.java:803 - Exception encountered during startup java.lang.RuntimeException: A node with address DM3PEPF00014567/10.115.155.147 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node.     at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:632) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:880) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.StorageService.initServer(StorageService.java:738) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.StorageService.initServer(StorageService.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395) [apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633) [apache-cassandra-3.11.10.jar:3.11.10]     at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786) [apache-cassandra-3.11.10.jar:3.11.10]
This node was up and running for last few months and suddenly started throwing this error. I can confirm that node with same IP address doesnt exists in cluster.

Here is the nodetool status for this DC :

When this happens I usually run nodetool removenode <host ID> command which will remove this node and then restart Cassandra on that cluster which will bring it up. But wondeirng how to fix this issue permanently as we are seeing this very intermittently.

Answer 1

The symptoms you described indicate to me that you have two nodes in the cluster configured with the same IP address that are trying to join the cluster. However, you didn't provide enough information so it's difficult to determine what is going.

A friendly note that you need to provide sufficient details when asking a question. The general guidance is that you (a) provide a good summary of the problem that includes software/component versions, the full error message + full stack trace; (b) describe what you've tried to fix the problem, details of investigation you've done; and (c) steps to replicate the problem.

For guidance, see Asking Good Cassandra Questions I published on the Apache Cassandra website. Additionally since you're new to DBA Stack Exchange, I would recommend taking the tour and reading the guide on how to ask questions. Cheers!

Answer 2

So I've seen this error before:

A node with address DM3PEPF00014567/10.115.155.147 already exists, cancelling join.
Use cassandra.replace_address if you want to replace this node.

If you're sure that two nodes aren't coming up with the same IP address, then it's likely that this IP address was used on one of the prior nodes, and needs to be removed from cluster gossip (system.peers table on all nodes). The easy way to do this, is to run a nodetool assassinate from any "good" node in the cluster.

nodetool assassinate 10.115.155.147

Then try to rejoin the node.