0

Symptoms

A new node being added to a Cassandra cluster is unable to gossip with seed nodes.

Another symptom is when an existing which has been restarted is unable to gossip with other nodes in the cluster reporting the following error during startup:

ERROR [main] 2019-08-15 18:46:32,241 CassandraDaemon.java:749 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) ~[apache-cassandra-3.11.4.jar:3.11.4]

In some cases, other nodes are able to see the affected node as operational but a problematic node is unable to gossip with other nodes. Here is a sample output of nodetool gossipinfo:

/10.1.2.4
  generation:0
  heartbeat:0
/10.1.2.3
  generation:0
  heartbeat:0
/10.1.2.6
  generation:1444263348
  heartbeat:6232
  ...
  DC:DC1
  STATUS:NORMAL,-1041938454866204344
  ...
/10.1.2.5
  generation:0
  heartbeat:0

One other symptom is that the affected node sees nodes in another DC as "down" (DN or Down/Normal) as shown in this sample nodetool status output:

Datacenter: r1 
============== 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns Host ID Rack 
DN 10.1.2.5 ? 256 9.0% 5279619a-550c-42b3-8150-61ad24f828f3 r1 
DN 10.1.2.3 ? 256 9.1% 5d1fa459-cdac-4658-b68d-c6e0933afcee r1 
DN 10.1.2.4 ? 256 10.5% a8f35c63-6a76-4e95-99f1-bef65d785366 r1 
Datacenter: DC1 
=============== 
Status=Up/Down 
|/ State=Normal/Leaving/Joining/Moving 
-- Address Load Tokens Owns Host ID Rack 
UN 10.1.2.6 18.9 GB 256 9.5% 36fdcf57-0274-43b8-a501-c0e475e3e30b RAC1

1 Answer 1

0

Cause

The most common reasons for gossip issues are:

  • lack of network connectivity,
  • clock drift, and
  • commit log permissions.

NETWORK - The gossip protocol is used by nodes to communicate information with each other in the cluster. If there is no connectivity between nodes, they will not be able to gossip with each other.

CLOCKS - Cassandra uses a server's local time to generate a "heartbeat generation". When gossiping with other nodes, Cassandra compares the local gossip generation with the other node as a way to validate the gossip information.

If the gossip generation is too far off, Cassandra will reject the other node's gossip info as invalid meaning that the gossip failed.

COMMIT LOG - If Cassandra does not have write permissions on the commitlog/ directory or files, memtable flushes will be blocked and prevent nodes from gossiping. For example, this can happen if Cassandra was accidentally started as root instead of cassandra or some other service account.

Solution

For gossip to work, nodes must be able to communicate with each other on TCP port 7000 (port 7001 for SSL) in both directions:

  • Check software firewall such as iptables or firewalld for misconfiguration.
  • Check for missed steps in your organisation's server provisioning process - did new security settings get inadvertently applied to the node?
  • Check ports on network devices for misconfiguration or hardware failure.
  • Check network policies such as quality-of-service (QoS) or bandwidth throttling rules for misconfiguration - do they apply to this environment?

Check the time on the servers. Make sure that NTP is configured and that there is no time-drift.

Finally, check the file permissions on the commitlog/ directory and reset ownership as appropriate. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.