0

We have three node cassandra cluster which has a set ttl of around 48 hours and automatic compactions enabled and also automatic incremental repair enabled every 24 hours. After doing a db purge[cleaning entire db] as soon as I start all services the repair percentage starts to go low very rapidly; and it ends up as 0% in less than a minute.

Any inputs on how we can try to find out who is corrupting the DB like this??

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address       Load       Tokens       Owns    Host ID                               Rack
UN  IP1  15.14 GiB  256          ?       <id1>  rack1
UN  IP2  13.66 GiB  256          ?       <id2>  rack1
UN  IP3  14.03 GiB  256          ?       <id3>  rack1

Note: Non-system keyspaces dont have the same replication settings, effective ownership information is meaningless
$ nodetool  info
ID                     : <id>
Gossip active          : true
Thrift active          : true
Native Transport active: true
Load                   : 13.66 GiB
Generation No          : 1710364981
Uptime (seconds)       : 748346
Heap Memory (MB)       : 5583.54 / 31129.63
Off Heap Memory (MB)   : 1034.91
Data Center            : datacenter1
Rack                   : rack1
Exceptions             : 4
Key Cache              : entries 14365, size 40.99 MiB, capacity 100 MiB, 23860 hits, 223259 requests, 0.107 recent hit rate, 14400 save period in seconds
Row Cache              : entries 0, size 0 bytes, capacity 0 bytes, 0 hits, 0 requests, NaN recent hit rate, 0 save period in seconds
Counter Cache          : entries 0, size 0 bytes, capacity 50 MiB, 0 hits, 0 requests, NaN recent hit rate, 7200 save period in seconds
Chunk Cache            : entries 18, size 1.12 MiB, capacity 480 MiB, 68466711 misses, 68719176 requests, 0.004 recent hit rate, 48.355 microseconds miss latency
Percent Repaired       : 0.0%
Token                  : (invoke with -T/--tokens to see all 256 tokens)

2
  • Percent Repaired at 0.0% is not necessarily an issue, and certainly not a sign of corruption - for reference I answered to a similar question here: stackoverflow.com/questions/78223728/… To better lay out the context, could you provide the Cassandra version, and explain what you mean by automatic compactions and "cleaning entire db"? Apr 3 at 9:54
  • Cassandra version is 3.11.3 and by automatic compactions I meant the minor compactions triggered by the compaction strategy by cassandra itself and by cleaning the entire db I meant dropping all non system keyspaces and deleting the directories for those keyspaces from the file system itself [from all three nodes] Apr 3 at 13:24

2 Answers 2

0

Percent Repaired at 0.0% is not a problem.

To understand the why, we need to dig a little into how incremental repair works.

Incremental repairs in Cassandra are meant to save repair overhead, by marking on every SStable (data files) metadata whether or not it was repaired. Previously repaired SStables are skipped from subsequent incremental repairs via the repaired status.

The Percent Repaired value in nodetool info amounts for the percentage of data marked as repaired on a given node, and that is extrapolated from the metadata across all SStables that a node stores.

Since all the data in non-system keyspaces was purged, all the SStables that were previously marked as repaired were removed from the Percent Repaired formula, resulting in the 0.0% value.

Importantly, incremental repairs in versions up to Cassandra 4 are buggy enough for me to recommend you running non-incremental repairs. To make matters worse, virtually every repair option in Cassandra 3 enforces incremental repairs.
Nonetheless, there are 2 ways to overcome that and have a sustainable repair strategy:

  • Run sub-range repairs - with --start-token and --end-token flags
  • Run datacenter-only repairs - with --in-dc flag

In order to minimize performance impact, work around incremental repairs, and facilitate repair schedule management The Last Pickle developed the open-source tool cassandra-reaper. It automates sub-range repair setup, and allows users to manage schedules, granular repairs, intensity, among other aspects, and I can't recommend it enough, even if you're running Cassandra 4.

0

The Percent Repaired being at 0% is not indicative of corruption and is not an issue at all.

When using incremental repairs, SSTables which have already been validated are marked as "repaired" (SSTable metadata repairedAt property is set to the time when repair occurred) so Cassandra doesn't waste resources repairing the SSTable again.

When a DROP or TRUNCATE is performed on a table (or keyspace), the SSTables which were previously marked as repaired no longer exists so effectively zero SSTables are repaired. It is therefore expected that the Percent Repaired metric has a zero value. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.