We are using Cassandra for our services with a cluster of 4 data center and 6 nodes in each DC. Recently we update our node SKU by first adding 6 new nodes in same data center, migrating the data from existing nodes to new nodes and then removed old nodes by running either nodetool decommission or nodetool removenode command.
After this we started seeing that new Cassandra nodes are frequently going down and throwing error A node with address already exists, cancelling join while trying to come up. It then doesnt come up. I need to remove the node which is in DN status and restart Cassandra service again to make this node up. But after few days same thing happens again.
Wondering why we started seeing this issue suddenly and how can this be fixed. We are using Apache Cassandra 3.11.10 version.
Error logs :
ERROR [main] 2023-04-17 17:28:29,128 CassandraDaemon.java:803 - Exception encountered during startup java.lang.RuntimeException: A node with address DM3PEPF00014567/10.115.155.147 already exists, cancelling join. Use cassandra.replace_address if you want to replace this node. at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:632) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:880) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:738) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.StorageService.initServer(StorageService.java:687) ~[apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:395) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:633) [apache-cassandra-3.11.10.jar:3.11.10] at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:786) [apache-cassandra-3.11.10.jar:3.11.10]
This node was up and running for last few months and suddenly started throwing this error. I can confirm that node with same IP address doesnt exists in cluster.
When this happens I usually run
nodetool removenode <host ID> command which will remove this node and then restart Cassandra on that cluster which will bring it up. But wondeirng how to fix this issue permanently as we are seeing this very intermittently.