Cassandra Read-Write Latency

Question

INFO  [HintsDispatcher:416] 2024-02-12 01:42:41,180 NoSpamLogger.java:91 - Maximum memory usage reached (536870912), cannot allocate chunk of 1048576

We are getting the above INFO in logs. Also, we are facing latency alerts (Condition: More than 2 seconds with 5 minutes span).

Is there any relation between latency and this INFO entry in the log?

An increase in file_cache_size_in_mb could help here anyway or the condition seems too tight and we can expand that (e.g. 5 seconds in 5 minutes span). No complaints about latency issues from the Application side as of now.

Answer 1

The message you shared relates to failed allocation to the chunk cache, which works as a Cassandra-native page cache to store SSTable chunks in memory.

While it is of no immediate concern, it indirectly relates to read latencies, but not to write latencies - the goal of the chunk cache is to speed up reads by allowing Cassandra to read chunks of data from memory off-heap rather from disk, which is generally faster - so each time you see this message it's a missed opportunity to speed up a read request by reading data from memory.

If you see the message you shared often, it means that the node can't continuously free up space in the chunk cache at the same rate that read requests demand new data to be allocated in it. In other words, increasing the chunk cache size as you considered may help, but only to the extent that you size it equal or greater than the volume of hot data being read beyond the reallocation capabilities of the nodes, which may be tricky if the clients cause intense read periods. This requires some trial and error to get it right.

If write latencies are also affected, I would look somewhere else. For high latencies in writes, the first places I'd look would be into garbage collection activity, iowait in the commit log drive, and CPU utilization, but such issues can come from multiple different sources.