0

A Cassandra cluster which for the most part is operational would sometimes experience service disruptions particularly when one (or more) node is unable to gossip with other nodes in the cluster.

Symptoms

One symptom is nodes randomly going up and down for no apparent reason. Here is an example extract from the system.log of a node:

INFO  [GossipTasks:1] 2016-04-29 02:47:32,559  Gossiper.java:1001 - InetAddress /10.1.2.3 is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:50:47,123  Gossiper.java:1001 - InetAddress /10.1.2.4 is now DOWN
INFO  [GossipTasks:1] 2016-04-29 02:54:59,640  Gossiper.java:1001 - InetAddress /10.1.2.5 is now DOWN
INFO  [SharedPool-Worker-2] 2016-04-29 03:01:23,828  Gossiper.java:987 - InetAddress /10.1.2.4 is now UP
INFO  [SharedPool-Worker-1] 2016-04-29 03:01:59,432  Gossiper.java:987 - InetAddress /10.1.2.5 is now UP
INFO  [SharedPool-Worker-7] 2016-04-29 03:02:01,839  Gossiper.java:987 - InetAddress /10.1.2.3 is now UP

Similarly, different nodes appear to be down in the nodetool status output depending on the node where the command was ran, for example:

Datacenter: Cassandra
=====================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address   Load       Tokens  Owns    Host ID                               Rack
DN  10.1.2.3  8.97 GB    256     ?       a50dfef5-229d-4d15-89d9-971bec01094b  rack1
UN  10.1.2.5  8.9 GB     256     ?       a16b71a2-9b95-4669-a6bd-d7326bd279e2  rack1
DN  10.1.2.4  9.09 GB    256     ?       ac01b6f9-3cb9-47ff-83c6-0404836386eb  rack1
UN  10.1.2.6  10.65 GB   256     ?       9c0ef3a2-aad7-4d06-b015-f32ddccac750  rack1

What could possibly be causing this issue?

1 Answer 1

0

Cause

There have been isolated reports of this issue in very small clusters. The symptoms described above has been identified to be related to clusters configured with GossipingPropertyFileSnitch (GPFS) but with a copy of cassandra-topology.properties (for PropertyFileSnitch) in the conf/ directory together with cassandra-rackdc.properties both used for defining the rack placement of nodes.

The combination of GPFS with cassandra-topology.properties can be verified in the startup messages log:

INFO  [main] 2013-02-17 15:31:26,039  GossipingPropertyFileSnitch.java:63 - \
  Loaded cassandra-topology.properties for compatibility

It is important to note that the issue is very intermittent and not all vectors which trigger the problem are known yet.

Workaround

By design, the GossipingPropertyFileSnitch falls back on the PropertyFileSnitch's cassandra-topology.properties as a means to allow clusters to be migrated to GPFS.

If the cluster is already on GossipingPropertyFileSnitch, check that cassandra-topology.properties has been removed or does not exist even if there are no issues with the nodes to ensure the cluster does not encounter problems in the future.

For more information, see CASSANDRA-11508. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.