I have cassandraCassandra 4.1.5 version installed on my DC and DR serves.. Below is the initial configuration I had -
On DC:
First DC
I have 3 nodes with IPs x.x.x.112, x.x.x.113 and x.x.x.114 where x.x.x.114 is seed node and is part of seed_providerseed_provider list in all 3 nodes. All 3 nodes have below configuration -
cassandra-rackdc.properties:
cassandra-rackdc.properties:
dc=dc1
rack=rack1
cassandra.yaml:cassandra.yaml:
num_tokens: 16
seed_provider:
- seeds: "x.x.x.114, x.x.y.114"
listen_address: x.x.x.112 (IP address corresponding node)
rpc_address: x.x.x.112 (IP address corresponding node)
endpoint_snitch: GossipingPropertyFileSnitch
On DR:
Second DC
I have 3 nodes with IPs x.x.y.112, x.x.y.113 and x.x.y.114 where x.x.y.114 is seed node and is part of seed_provider list in all 3 nodes. All 3 nodes have below configuration - cassandra-rackdc.properties:
cassandra-rackdc.properties:
dc=dr1
rack=rack1
cassandra.yaml:cassandra.yaml:
num_tokens: 16
seed_provider:
- seeds: "x.x.x.114,x.x.y.114"
listen_address: x.x.y.112 (IP address corresponding node)
rpc_address: x.x.y.112 (IP address corresponding node)
endpoint_snitch: GossipingPropertyFileSnitch
Due to some issues on network and infra, I had to remove the DR nodes using nodetool removenode and then I reinstalled and restarted the DR nodes. Post that I found that DR nodes are not connecting. Also, I removed DC and DR seed nodes from seed_providers properties of both clusters. I can see DR nodes in gossipinfogossipinfo on DC and one of the node is in removed status and couple of nodes are in removed with REMOVAL_COORDINATOR where this coordinator node is not in cluster.
My custom schema is as of now as CREATE KEYSPACE mykeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3'} AND durable_writes = true; and this way I am able to serve from DC. But I want to setup my DR back now and want to change the keyspace as ALTER KEYSPACE mykeyspace WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '3', ''dr1': '3'} AND durable_writes = true;.
I have tried restarting nodes and cleaning data folder in DR but somehow DR nodes are behaving abnormal. One of node i.e. x.x.y.114 is automatically getting connected to DC nodes and its coming in nodetool status even there is no cross reference in DC and DR. And if I do nodetool status on DR node i.e. x.x.y.112, I can see DC nodes in DN status but x.x.y.114 is not part of it.
Is there anyway I can clean DR completely and add these nodes back as DR.
