Cassandra has stopped producing log files

Question

I've recently taken up the role of managing a Cassandra Cluster that has been running in production for a few years. This is my first time working with Cassandra so I would appreciate any insights.

It is a 3 node cluster with 100% replication across each node. The nodes each have a load of around 5 TB. The config is not ideal but they are working fine for now. However whenever there is an issue it is hard to troubleshoot since I noticed log files have stopped updating a few years back.

I have made sure the logging directory was not changed, and all the logging levels are at default. Memory and disk are still at reasonable usage levels. I noticed each of the logs have reached 10 zip files and stopped producing anything. Other than that I do not see the issue. I would appreciate the help.

Thanks

Answer 1

It's highly unusual for messages to not have been logged for a few years. There is a possibility that logs are directed to a directory other than what you are checking.

Here are a list of things to check:

• On one of the nodes, verify which log file the Cassandra process is writing to using Linux utilities such as lsof.
• You didn't specify which Cassandra version is running or which configuration file(s) you checked but if you're running an old version of Cassandra, logging got switched from Log4j to Logback and you might not be looking at the correct configuration.
• The Cassandra process might not have write access to the logs. This can happen if Cassandra was accidentally started as the root user so check permissions on the directory and files.
• Check that the mount point for the log directory has free space available.

This should hopefully give you enough to investigate. Cheers!