1

We currently have a setup that consists of the following:

  • Cassandra 3.11.13
  • Datacenter 1 = 3 Cassandra nodes using LOCAL_QUORUM and replication factor 3
  • Datacenter 2 = 3 Cassandra nodes using LOCAL_QUORUM and replication factor 3

I understand that there are a lot of metrics that can be monitored (based on this - https://cassandra.apache.org/doc/latest/cassandra/operating/metrics.html), but what is not clear to me (at least) is how we can monitor cross datacenter replication latency. From the link, there are several latency metrics available, but it seems like these are for the local node or local datacenter cluster and not for cross datacenter replication.

In my searching, I've found discussions such as this https://issues.apache.org/jira/browse/CASSANDRA-11569 which suggest that there is something in Cassandra for this, but again, it's not clear how we are supposed to get these metrics out.

So, my question is simply, is how do we monitor this? Is there something built into Cassandra to help with this, or do we need to implement some other tool to check this?

Thanks in advance for any help on this!

2 Answers 2

1

There isn't a metric that tracks cross-DC replication latency in Cassandra.

The closest you can get is the cross-DC messaging latency which measures how long it took for a message from a node in the source DC to reach a node in the destination DC. To be clear, this metric applies to all messages, not just those for replication.

This is the metric added by CASSANDRA-11569 which you quoted in your question. The Mbean details are:

org.apache.cassandra.metrics:type=Messaging,name={dc}-Latency

Note that the metric name is the DC's name + -Latency suffix. For example, if the DC's name is DC1 then the metric's name is DC1-Latency.

For details, see MessagingMetrics.java in C* 3.11.13. Cheers!

1
  • Hmm, I must have got the syntax wrong when trying to access this before, but now it works, so that is something. I assume (based on what I'm seeing) that the data presented from this mbean is for "all time"; well until the process is restarted? It also seems like it's per process too, when I check "DC1-Latency" on each host, I seem to get different values. What would be the best way to interpret these in order to conclude if there may be latency between the datacenters? Mar 2, 2023 at 16:09
1

Just to expand on Erick's answer, there was a similar question that I answered on Stack Overflow. It might help you know what to look for: https://stackoverflow.com/questions/75309210/cassandra-inter-cluster-latency-metrics/75315219#75315219

So, my question is simply, is how do we monitor this? Is there something built into Cassandra to help with this, or do we need to implement some other tool to check this?

You will need a monitoring stack to make this work. Many teams use Grafana to monitor Cassandra, and there is a plugin available to make that work: https://grafana.com/grafana/plugins/hadesarchitect-cassandra-datasource/

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.