My team runs Cassandra for an internal application. This is our setup:
- We are running Cassandra version 3.11.13.
- We have multiple keyspaces; each microservice has its own.
- We have an active-active architecture in two AWS regions.
- Each region has 4 nodes: 2 seeds and 2 members.
- The topology is a replication factor of 4 in each datacenter/region, so all nodes have 100% of the data.
- Our read consistency is LOCAL_QUORUM and our serial consistency is LOCAL_SERIAL, in order to maintain the active-active status.
We recently had an incident where we were seeing read timeouts during queries from the application client. They seemed to be correlated with only one keyspace. After much troubleshooting, we found that a
nodetool repair fixed the issue. We did not have any exceptions in the log files to lead us to understand that there was data inconsistency or corruption, so we can only assume this was the case because it was fixed by a repair. There were also no EC2 hardware or OS system failures that would indicate disk issues. As a result, we have no real root cause, and don't know how to prevent this in the future.
One symptom we did notice was the following log message that kept repeating for that particular keyspace:
INFO [Native-Transport-Requests-1] 2023-07-04 11:53:15,913 MigrationManager.java:286 - Update Keyspace '[REDACTED]' From KeyspaceMetadata ......
This symptom went away after the repair.
We are wondering if anyone has insight into what might have occurred, and subsequently, how to prevent it.
It may be worth noting that we were not running frequent enough repairs according to the guidelines. We were doing them approximately every three months, but the recommendation is to match the
gc_grace_seconds value you're using, which is currently 10 days.
It also might be important to know that we suspect there was a higher degree of load at the time of the incident due to a batch job we were running periodically, but we did not see signs of stress on the Cassandra nodes.