2 of 3 Cassandra nodes reporting each other as down

Question

I got this issue most probably due to 2-3 restarts I did for cluster where I restarted all nodes. Please see cluster status on each node. One node showing all 3 nodes up. Other two nodes are reporting each other as down. There is no Gossip issue as per logs and I verified 7000 port using telnet as well from both machines and telnet connecting to port.

Another thing is that now i am unable to connect to 10.0.81.154 using cqlsh. Also this node system logs are repeating following warnings after every 5 seconds:

WARN  [OptionalTasks:1] 2022-12-20 13:37:49,069 CassandraRoleManager.java:343 - CassandraRoleManager skipped default role setup: some nodes were not ready
INFO  [OptionalTasks:1] 2022-12-20 13:37:49,069 CassandraRoleManager.java:369 - Setup task failed with error, rescheduling

There is no gossip issue as per nodetool as well. Results are as follow:

root@ip-10-0-81-154:/home/ubuntu# nodetool -h ::FFFF:127.0.0.1 gossipinfo
/10.0.59.101
  generation:1671531617
  heartbeat:18904
  STATUS:22:NORMAL,-1277676488711986166
  LOAD:18866:1.28678324E9
  SCHEMA:16:d7e13e0d-5d66-3891-80ee-7541e185eef7
  DC:12:OSS-dc0
  RACK:14:eu-west-2b
  RELEASE_VERSION:6:4.0-beta2
  INTERNAL_IP:10:10.0.59.101
  RPC_ADDRESS:5:10.0.59.101
  NET_VERSION:2:12
  HOST_ID:3:83e77b0e-6c25-4163-8ebe-9d8647135f47
  RPC_READY:44:true
  INTERNAL_ADDRESS_AND_PORT:8:10.0.59.101:7000
  NATIVE_ADDRESS_AND_PORT:4:10.0.59.101:9042
  STATUS_WITH_PORT:21:NORMAL,-1277676488711986166
  TOKENS:20:<hidden>
/10.0.81.154
  generation:1671540728
  heartbeat:9533
  STATUS:22:NORMAL,-1185175657542272865
  LOAD:9508:1.286367011E9
  SCHEMA:16:d7e13e0d-5d66-3891-80ee-7541e185eef7
  DC:12:OSS-dc0
  RACK:14:eu-west-2c
  RELEASE_VERSION:6:4.0-beta2
  INTERNAL_IP:10:10.0.81.154
  RPC_ADDRESS:5:10.0.81.154
  NET_VERSION:2:12
  HOST_ID:3:d7805f77-2165-489e-b43b-728379c50e44
  RPC_READY:34:true
  INTERNAL_ADDRESS_AND_PORT:8:10.0.81.154:7000
  NATIVE_ADDRESS_AND_PORT:4:10.0.81.154:9042
  STATUS_WITH_PORT:21:NORMAL,-1185175657542272865
  TOKENS:20:<hidden>
/10.0.15.45
  generation:1671531646
  heartbeat:18909
  STATUS:22:NORMAL,-1207378899666305813
  LOAD:18871:1.289545177E9
  SCHEMA:16:d7e13e0d-5d66-3891-80ee-7541e185eef7
  DC:12:OSS-dc0
  RACK:14:eu-west-2a
  RELEASE_VERSION:6:4.0-beta2
  INTERNAL_IP:10:10.0.15.45
  RPC_ADDRESS:5:10.0.15.45
  NET_VERSION:2:12
  HOST_ID:3:3bd9ae5d-5fad-468f-9e33-98111fa59fe5
  RPC_READY:34:true
  INTERNAL_ADDRESS_AND_PORT:8:10.0.15.45:7000
  NATIVE_ADDRESS_AND_PORT:4:10.0.15.45:9042
  STATUS_WITH_PORT:21:NORMAL,-1207378899666305813
  TOKENS:20:<hidden>

Other nodetools shows similar status i.e. on .45 node all good but other two nodes report each other as down. All nodes are with good cpu and memory i.e. 4vcpu and 16 GB ram and not much data in there for now. ESB volumes are with 3000 iops and ample disk space.

There is no tombstone and compaction issue. Both nodes are down for different reason. We suspect that once we will resolve node .154 issue i.e. skipped default role setup: some nodes were not ready then that may help other node to come up as well.

Also if I try nodetool repair command on each of down node then they give error that unable to connect to ip:7000 port of other node despite the fact that telnet works good and no gossip error in logs. Nodetool repair works good on .45 node which is showing all nodes up

Answer 1

The first thing I would check is whether the time is set correctly on all nodes. If the system time on a node has drifted, it would prevent nodes from gossiping with each other. It is very important that nodes have (1) NTP installed, and (2) NTP running correctly.

Second is to make sure that the Cassandra process has the correct read/write permissions to commit logs. When the Cassandra process has no access to the commit logs, this can prevent gossip from working. For example, if the Cassandra process is supposed to run as the cassandra user but the files are owned by root then cassandra won't have permission on the logs.

Third, I've noted that the nodes are running with a beta version of Cassandra (4.0-beta2):

  RELEASE_VERSION:6:4.0-beta2

None-GA versions are not intended for general use -- they are intended for pre-release testing only. We recommend that you upgrade to the latest version of Cassandra 4.0 to make sure that your cluster doesn't hit any known issues with gossip which have already been fixed. Cheers!

---

👉 Please support the Apache Cassandra community by hovering over the cassandra tag then click on the Watch tag button. 🙏 Thanks!

Answer 2

In addition to Erick's answer, I'd suggest a couple of things:

There is no gossip issue as per nodetool as well.

Remember that the results of gossip and Nodetool are unique to each node. Just because nodetool gossipinfo looks good on one node, doesn't mean that it looks good on all nodes.

Judging by the nodetool status output, it almost looks like regions 2b and 2c can't talk to each other. Maybe try creating another node in each region and see if they can join ok. If they can't, then verify that the cloud (AWS?) security group config is set up properly.

Also check the system.peers table on the one good node, and make sure that it only contains IPs for the other two nodes. If it has other, old IPs, they might be causing a conflict. Might also be a good idea to check the system.peers tables of the two troublesome nodes, and just make sure that the IPs present in that table actually belong there (should only be 2).