3

Not sure if this is bug with cassandra 4.0.5 but when i remove data directory and clean up all other directory and try to join node to cluster i get below error. 4.0.4 seem to be works fine

DEBUG [main] 2022-09-09 09:05:57,351 StorageService.java:2600 - Node l2r1cassan002.abc.xyz.com/10.11.180.70:7600 state bootstrapping, token [7440677904722156112, 202341339740687413, -4608008619501212021, 4549633523338081274, -7205177367714369394, -1733332418442686603, 2714110185983303464, -8795181920228268032, 6361305097274554454, -2829374318677624892, -5586154072017326096, 1642609569643270651, 8684197910402748084, -642035307750474343, 3731961590725730479, 5625389404563660813]
INFO  [main] 2022-09-09 09:05:57,352 StorageService.java:1634 - JOINING: sleeping 30000 ms for pending range setup
ERROR [main] 2022-09-09 09:06:27,354 CassandraDaemon.java:911 - Exception encountered during startup
java.lang.IllegalStateException: Unable to contact any seeds: [/10.11.180.69:7600, /10.11.180.71:7600, /10.11.227.152:7600]
        at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1766)
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1058)
        at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1019)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:801)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:731)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:420)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
DEBUG [StorageServiceShutdownHook] 2022-09-09 09:06:27,359 StorageService.java:1636 - DRAINING: starting drain process
INFO  [StorageServiceShutdownHook] 2022-09-09 09:06:27,360 HintsService.java:222 - Paused hints dispatch
DEBUG [StorageServiceShutdownHook] 2022-09-09 09:06:27,361 StorageService.java:2399 - Ignoring application state RPC_READY from l2r1cassan002.abc.xyz.com/10.11.180.70:7600 because it is not a member in token metadata
WARN  [StorageServiceShutdownHook] 2022-09-09 09:06:27,361 Gossiper.java:2032 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
Improve this question

2 Answers 2

Reset to default
1

I'm not aware of any changes in Cassandra 4.0.5 over 4.0.4 that would cause this problem.

In my experience, the most common causes of nodes unable to gossip are:

  • Network issue where there's no connectivity between nodes on the gossip port. Check that the firewall is not blocking internode communication.
  • The clocks are out of sync. Make sure that NTP is installed and working as expected.
  • File(s) in the commitlog/ directory are owned by root preventing the Cassandra process from writing new files.

If you think none of this applies, please post detailed steps that replicate the issue and I'd be happy to review. Cheers!

Improve this answer
6
  • Hi Erick, Thanks for reply. I will check and reply back What i did was, stopped service, deleted all files under data, commitlog and tried to restart service. Once i restart, i am getting error as above. Let me check on time sync and get back to you
    – Narayana Swamy
    Commented Sep 10, 2022 at 18:23
  • Hi Eric Yes.. there is 5mins diffrence between nodes. I was able to join to cluster once i syned time on all nodes. Thanks
    – Narayana Swamy
    Commented Sep 12, 2022 at 3:10
  • That's awesome! 👍
    – Erick Ramirez
    Commented Sep 12, 2022 at 3:29
  • Hi Erick This it hapened again. I have 3+3 nodes (DC1 3 and DC2 3) i was not able to do CQLSH, it said it was not able to contact host. So i restarted. After i restart, i am not able to join to clusrer. I checked time stamp, but did nto find any diffrence I am not able even after clearing data and commit log On Seed node i have this error DEBUG [GossipStage:1] 2022-09-16 13:45:06,393 StorageService.java:2342 - Ignoring state change for dead or unknown endpoint: /10.11.11.12:7600 INFO [Messaging-EventLoop-3-3] 2022-09-16 13:45:07,208
    – Narayana Swamy
    Commented Sep 16, 2022 at 13:48
  • OutboundConnection.java:1150 - /10.11.11.11:7600(/10.11.11.11:39794)->/10.11.11.12:7600-URGENT_MESSAGES-90ada048 successfully connected, version = 12, framing = CRC, encryption = unencrypted DEBUG [GossipStage:1] 2022-09-16 13:46:05,253 StorageService.java:2342 - Ignoring state change for dead or unknown endpoint: /10.11.11.12:7600 DEBUG [CompactionExecutor:4] 2022-09-16 13:46:06,045 TimeWindowCompactionStrategy.java:129 - TWCS skipping check for fully expired SSTables
    – Narayana Swamy
    Commented Sep 16, 2022 at 13:49
-1
  1. I don't know why one of the nodes went out of sync.
  2. Like @Eric said, check for time sync, permissions
  3. Make sure you delete data directory and commit logs
  4. Restart service
  5. If you see

java.lang.IllegalStateException: Unable to contact any seeds on bad node and if you see

DEBUG [GossipStage:1] 2022-09-16 15:51:44,615 StorageService.java:2342 - Ignoring state change for dead or unknown endpoint on seed node

use nodetool assasinate <IP Address>.

Nodetool assasinate worked for me.

Improve this answer
1
  • This is not a valid answer. The assassinate command is for solving a completely different issue. It indicates that there's another problem in your environment and you hacked it by running assassinate but it is not the right way of dealing with it in general.
    – Erick Ramirez
    Commented Sep 16, 2022 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.