5

I'm running Cassandra on three nodes. Here's their nodetool status output:

ubuntu@ip-10-0-8-8:~$ nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  10.0.9.8   2.07 MB    256     ?       c8d574b9-540c-410f-9326-789eb75d3d14  1c
UN  10.0.8.8   2.06 MB    256     ?       d9454056-a358-4428-ab5f-c03e8042167e  1d
UN  10.0.10.8  2.01 MB    256     ?       3617643d-b0a8-4b72-a9d4-feded4445292  1a

ubuntu@ip-10-0-9-8:~$ nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  10.0.9.8   2.07 MB    256     ?       c8d574b9-540c-410f-9326-789eb75d3d14  1c
UN  10.0.8.8   2.06 MB    256     ?       d9454056-a358-4428-ab5f-c03e8042167e  1d
DN  10.0.10.8  2.09 MB    256     ?       3617643d-b0a8-4b72-a9d4-feded4445292  1a

ubuntu@ip-10-0-10-8:~$ nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address    Load       Tokens  Owns    Host ID                               Rack
UN  10.0.9.8   2.07 MB    256     ?       c8d574b9-540c-410f-9326-789eb75d3d14  1c
UN  10.0.8.8   2.06 MB    256     ?       d9454056-a358-4428-ab5f-c03e8042167e  1d
UN  10.0.10.8  2.01 MB    256     ?       3617643d-b0a8-4b72-a9d4-feded4445292  1a

Everything looks fine except one thing (last line in second block):

DN  10.0.10.8  2.09 MB    256     ?       3617643d-b0a8-4b72-a9d4-feded4445292  1a

The D in the start of the line indicates the node being down. How can it be that 10.0.9.8 is seeing the node as down while the other nodes are seeing it just fine? Does this lead to inconsistencies?

Using Cassandra version 2.1.1 by the way.

3

Running nodetool enablegossip on the host that appeared down to other nodes fixed it for me and for now. However, it appeared as down to all other nodes I checked. Running in a non-cloud environment.

I was curious what my other nodes said and I found one that had the same issue (like your 10.0.9.8, showing 10.0.10.8 as down). Only running nodetool enablegossip on 10.8 didn't help. But running disablegossip first and then enablegossip again did!

2

The gossip protocol is responsible for keeping nodes informed of the states of all the other nodes. I have seen it in the past where one node (for whatever reason) sees a node as "DN" while others do not. Typically this happens in cloud environments, and is usually caused by network events or inconsistencies.

Does this lead to inconsistencies?

Yes! The nodes which see 10.0.10.8 as "DN" will not replicate data to it. And assuming that it has been this way longer than 3 hours, they will have stopped logging hints for it as well.

For the quick solution, I would bounce the Cassandra process on both 10.0.10.8 and 10.0.9.8. As it comes back, tail the system.log file and make sure that it properly connects with all other nodes.

If that doesn't do it, try setting your phi_convict (cassandra.yaml) to 10 or 12 if you're in the cloud (on all nodes). That will give your nodes a little more tolerance to being marked as "DN" due to network latency.

Another thing to look at, is to double-check your cloud availability zone for the problem node(s). It's possible that nodes in one cloud AZ might behave differently network-wise. So if you find that these issues are a particular to an AZ, you may want to contact your cloud provider about that.

If it still doesn't come back, wipe the node's data and re-bootstrap it. Note, that when you do get your cluster back to full "UN," that you should run a repair on the node that was seen as "DN."

Using Cassandra version 2.1.1

And I would upgrade this immediately. If you're stuck on the 2.1 version, you should at least be on the latest patch release (for bug fixes) of 2.1.18. But there's been a lot of fixes an improvements since 2.1.1, which could be contributing to this issue.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.