0

Nodetool describecluster shows unreachable nodes even though the nodes are UN. Netstats output shows the connection established with port 7001 for the nodes which are shown unreachable. Cassandra logs don’t have any errors except for below messages.

INFO - Finished hinted handoff of file xx-xx—xx-1.hints to endpoint /xxx.xxx.xxx partially.

Can you please help me resolve the issue

1
  • Is that the exact message? Commented Feb 21 at 6:47

1 Answer 1

1

The most common reasons nodes show up as "unreachable" are:

  • bad network, e.g. no route between nodes, firewalls blocking traffic between nodes, intermittent connectivity issues;
  • nodes pausing due to "noisy neighbours" in a virtual environment, e.g. a large VM co-hosted with Cassandra node is stealing CPU cycles from smaller nodes;
  • JVM getting swapped out leading to nodes pausing;
  • nodes overloaded with app requests leading to longer GC pauses.

The first two are easy to rule out with the help of network and sysadmins.

With the third, you need the help of a sysadmin to make sure that swap is disabled in the operating system.

In my experience, overloaded nodes is the most common reason nodes have issues gossiping with each other so they are reported as "unreachable" even Cassandra in up on the node. You'll need to review the logs further for evidence of either (a) long GC pauses, or (b) lots of short GC pauses which when added up together equate to long pauses.

Consider adding more nodes to your cluster to increase its capacity. Your cluster should be sized for peak traffic, not average throughput. Cheers!

1
  • Thanks for the response, I will debug on the four reasons mentioned and try to find the cause for this.
    – Solver
    Commented Feb 26 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.