Problem with Cassandra data corruption/inconsistency

Question

My team runs Cassandra for an internal application. This is our setup:

• We are running Cassandra version 3.11.13.
• We have multiple keyspaces; each microservice has its own.
• We have an active-active architecture in two AWS regions.
• Each region has 4 nodes: 2 seeds and 2 members.
• The topology is a replication factor of 4 in each datacenter/region, so all nodes have 100% of the data.
• Our read consistency is LOCAL_QUORUM and our serial consistency is LOCAL_SERIAL, in order to maintain the active-active status.

We recently had an incident where we were seeing read timeouts during queries from the application client. They seemed to be correlated with only one keyspace. After much troubleshooting, we found that a nodetool repair fixed the issue. We did not have any exceptions in the log files to lead us to understand that there was data inconsistency or corruption, so we can only assume this was the case because it was fixed by a repair. There were also no EC2 hardware or OS system failures that would indicate disk issues. As a result, we have no real root cause, and don't know how to prevent this in the future.

One symptom we did notice was the following log message that kept repeating for that particular keyspace:

INFO  [Native-Transport-Requests-1] 2023-07-04 11:53:15,913 MigrationManager.java:286 - Update Keyspace '[REDACTED]' From KeyspaceMetadata ......

This symptom went away after the repair.

We are wondering if anyone has insight into what might have occurred, and subsequently, how to prevent it.

It may be worth noting that we were not running frequent enough repairs according to the guidelines. We were doing them approximately every three months, but the recommendation is to match the gc_grace_seconds value you're using, which is currently 10 days.

It also might be important to know that we suspect there was a higher degree of load at the time of the incident due to a batch job we were running periodically, but we did not see signs of stress on the Cassandra nodes.

Answer 1

correlated with only one keyspace

Which means that it's probably due to the write pattern of one particular application.

not running frequent enough repairs

This all depends on the required consistency. If the data doesn't have to be 100% correct or up-to-date, then repairs probably are more trouble than they are worth. But, if higher levels of consistency are required, then running repair once per week is usually a good idea. This can be set up to be easily managed by something like Cassandra Reaper.

higher degree of load at the time of the incident due to a batch job we were running periodically

This is a big indicator to me. Even though the nodes didn't appear to be stressed, there are a lot of reasons why replication can fail. The usual cause is too much write back-pressure was built up, causing one or more nodes to drop incoming writes.

Network inconsistency is also a big cause, so that might be something to look at, too.

don't know how to prevent this in the future

Multi-tenant clusters are problematic with Cassandra, because it makes it difficult to find root causes (like you are seeing) and then do anything about it without affecting other applications. It might be worthwhile to split-out this application from the rest and give it its own cluster. That way you could scale the resources to better-fit its write patterns.

I would also try having a look at the application or batch job running on the keyspace in question. If the write throughput can be throttled back, it should help things. This can be done by either limiting the in-flight write threads in code. The other option, is to have the batch job point to something like a Pulsar or Kafka topic, and then build a consumer to read from the topic and apply the writes to Cassandra.

Also check the disk IO on each node. If it's maxed out, that might be where the bottleneck is. In which case, that could be fixed by selecting a different drive type for the instance.