Issue with re-adding DR nodes

Question

I have Cassandra 4.1.5 installed.. Below is the initial configuration:

First DC

I have 3 nodes with IPs x.x.x.112, x.x.x.113 and x.x.x.114 where x.x.x.114 is seed node and is part of seed_provider list in all 3 nodes.

cassandra-rackdc.properties:

dc=dc1
rack=rack1

cassandra.yaml:

num_tokens: 16
seed_provider:
    - seeds: "x.x.x.114, x.x.y.114"
listen_address: x.x.x.112 (IP address corresponding node)
rpc_address: x.x.x.112 (IP address corresponding node)
endpoint_snitch: GossipingPropertyFileSnitch

Second DC

I have 3 nodes with IPs x.x.y.112, x.x.y.113 and x.x.y.114 where x.x.y.114 is seed node and is part of seed_provider list in all 3 nodes.

cassandra-rackdc.properties:

dc=dr1
rack=rack1

cassandra.yaml:

num_tokens: 16
seed_provider:
     - seeds: "x.x.x.114,x.x.y.114"
listen_address: x.x.y.112 (IP address corresponding node)
rpc_address: x.x.y.112 (IP address corresponding node)
endpoint_snitch: GossipingPropertyFileSnitch

Due to some issues on network and infra, I had to remove the DR nodes using nodetool removenode and then I reinstalled and restarted the DR nodes. Post that I found that DR nodes are not connecting. Also, I removed DC and DR seed nodes from seed_providers properties of both clusters. I can see DR nodes in gossipinfo on DC and one of the node is in removed status and couple of nodes are in removed with REMOVAL_COORDINATOR where this coordinator node is not in cluster.

My custom schema is as of now as CREATE KEYSPACE mykeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3'} AND durable_writes = true; and this way I am able to serve from DC. But I want to setup my DR back now and want to change the keyspace as ALTER KEYSPACE mykeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3', ''dr1': '3'} AND durable_writes = true;.

I have tried restarting nodes and cleaning data folder in DR but somehow DR nodes are behaving abnormal. One of node i.e. x.x.y.114 is automatically getting connected to DC nodes and its coming in nodetool status even there is no cross reference in DC and DR. And if I do nodetool status on DR node i.e. x.x.y.112, I can see DC nodes in DN status but x.x.y.114 is not part of it.

Is there anyway I can clean DR completely and add these nodes back as DR.

Answer 1

You have provided limited detail so I don't fully understand why you removed the nodes in the first place but it is the wrong thing to do if your intention is to keep those nodes. Instead, you should have focused on resolving the underlying issue with the network.

When a node is removed from the cluster, it is placed into "quarantine" for 72 hours (3 days) to prevent the node from accidentally re-joining the cluster, for example, if an operator inadvertently restarts Cassandra.

A removed node will not be able to gossip with other nodes. It will stay in quarantine for 72 hours and cannot be added back to the cluster until the 3-day quarantine period has expired.

To override the 3-day quarantine period, you will need to set the system property very_long_time_ms on the command line with -Dcassandra. very_long_time_ms=<number_in_milliseconds> on ALL nodes which will require a restart. You will then need to restart the nodes again withOUT this setting to reset it to the default value. Cheers!