Newly added nodes showing as DN to each other in nodetool status output

Question

I have recently installed Cassandra 4.1.1 in 5 node onPremise Cluster for my production environment.. For testing purpose I just added two seed nodes in Config file and restarted the Server and service below is the status of Nodetool after that

In B server

Datacenter: datacenter1
=======================
>Status=Up/Down\
|/ State=Normal/Leaving/Joining/Moving\
--  Address      Load        Tokens  Owns (effective)  Host ID                               Rack\
DN  XXX.XX.XX.A  208.51 KiB  16      100.0%            8b141431-7421-4d13-919d-d88a4fea7649  rack1\
UN  XXX.XX.XX.B  239.76 KiB  16      100.0%            0025f537-35c8-48ba-92b3-cfdfaee78a99  rack1

In A server

Datacenter: datacenter1
=======================
>Status=Up/Down\
|/ State=Normal/Leaving/Joining/Moving\
--  Address      Load        Tokens  Owns (effective)  Host ID                               Rack\
UN  XXX.XX.XX.A  208.51 KiB  16      100.0%            8b141431-7421-4d13-919d-d88a4fea7649  rack1\
DN  XXX.XX.XX.B  239.76 KiB  16      100.0%            0025f537-35c8-48ba-92b3-cfdfaee78a99  rack1

Dont Know how to fix this and Now server has no data am about to load it..

Below are the GossipInfo status

nodetool gossipinfo
/XXX.XX.XX.B
  generation:1684942657
  heartbeat:1319
  LOAD:1313:213518.0
  SCHEMA:13:54e17321-3f2e-37ca-9b08-d91ba7bdd369
  DC:9:datacenter1
  RACK:11:rack1
  RELEASE_VERSION:6:4.1.1
  NET_VERSION:2:12
  HOST_ID:3:8b141431-7421-4d13-919d-d88a4fea7649
  RPC_READY:31:true
  NATIVE_ADDRESS_AND_PORT:4:XXX.XX.XX.B:9042
  STATUS_WITH_PORT:1213:NORMAL,-2221626050107619877
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:1212:<hidden>
/XXX.XX.XX.A
  generation:1684940942
  heartbeat:2950
  STATUS:2908:NORMAL,-1567249714150872921
  LOAD:2928:243794.0
  SCHEMA:13:54e17321-3f2e-37ca-9b08-d91ba7bdd369
  DC:9:datacenter1
  RACK:11:rack1
  RELEASE_VERSION:6:4.1.1
  RPC_ADDRESS:5:XXX.XX.XX.A
  NET_VERSION:2:12
  HOST_ID:3:0025f537-35c8-48ba-92b3-cfdfaee78a99
  RPC_READY:31:true
  NATIVE_ADDRESS_AND_PORT:4:XXX.XX.XX.A:9042
  STATUS_WITH_PORT:2907:NORMAL,-1567249714150872921
  SSTABLE_VERSIONS:7:big-nb
  TOKENS:2906:<hidden>

Kindly help me out in this regard

Answer 1

The down/normal (DN) status is an indication that the nodes cannot gossip with each other.

If there is a firewall in between them, make sure that bi-directional TCP traffic on the gossip port (default is 7000) is open.

Otherwise, you will need to do some investigation as to why they are showing as DN by reviewing the logs. Without knowing the reason, it will be hard for anyone to help you. Cheers!