Cassandra cluster monitoring

Question

We using Nagios for our servers monitoring.

Each web-environment have 1 cluster as backend - Cassanra 2.0. Each cluster have 3-5 nodes.

Question is: I want write a plugin for Nagios, for Cassandra's nodes monitoring. Unfortunately - I'm not so good familiar with Cassandra, and don't sure - what parameters need exactly to be checked?

I'm planing use Nodetool utility to grab data from nodes - but it have a lot of commands, and each provides lot of information (e.g. - cfstats, info, status etc).

So - for monitoring I need get some data about memory each node's usage, used disk space, may be - something else?

Answer 1

1. You can use check Java Heap memory for each node. Like Total Java Heap Memory and Using Java Heap Memory.
2. One most important is CPU Utilization for each node.
3. Set alerts for errors. system.log have lots of information about.
4. You may set alerts for data disk and log disk.
5. Heartbeat check of servers like if you dont receive in few min it will get alerts to you.
6. Also dropped mutations and hinted hindoff clear alerts.

Basically you need to start observe system.log and will get more and more error for monitoring.

Answer 2

In my clusters, I use jolokia so I can access JMX interface using HTTP. This way I can create scripts with Python that monitor Cassandra and send alerts to Nagios.

What you should monitor:

1. Java HEAP and GC
2. Compactions pending
3. Flushes pending
4. Hinted Handoff pending
5. Gossip Tasks pending
6. Read / Write latency
7. Key cache hit rate
8. Down end point count

Monitoring those have saved my cluster a couple times :)

I run 2 clusters, one in my day job, it has 4 nodes with 3 TB data (2 more nodes coming soon) and in my own company a small 3 node cluster.