1

We have encountered with an issue that 3 of our 5 nodes cluster is not dropping expired sstables automatically.Other 2 nodes its being dropped successfully.

  • We moved from STCS to TWCS compaction dated 9 th Aug with compaction window 1 day,GC_GRACE_SECONDS 10 seconds and TTL 30 days.
  • 23rd Aug team found some issue where call details records were not being processed,that time they restarted all cassandra nodes. Unfortunately no logs available for that day
  • Files being removed successfully for all 5 nodes since 9th Sep.
  • Since 24th Sep automatic dropping stopped for 3 nodes.Other 2 were dropping expired sstable.

Team checked undropped sstable with sstablemetadata and sstabledump but all data looked expired and tsstamp also showing current date /time. "expired" : true for all data found in sstabledump

Kindly help with troubleshooting method as old logs are unavailable

will nodetool garbagecollect help here for TWCS ? or it will mix and merge sstables?

Improve this question
1
  • Hi Team, Few sstables in problamtic node have minClustringValues 2023-08-23. The same date when restart happens and there is huge gap in date between minClustringValues and maxClustringValues. Is that the reason,sstables are not being dropped automatically for these nodes ? For Nodes where files are dropping correctly doesn't have minClustringValues 2023-08-23 and gap in date between minClustringValues and maxClustringValues is just 2-3 days
    – Cassandra Thrift
    Commented Oct 4, 2023 at 13:20

1 Answer 1

Reset to default
1

Usually when this happens, it's caused by some out-of-order write or overlap in time windows. Do you happen to use unsafe_aggressive_sstable_expiration? This can be set as a jvm option as well as in the table's compaction strategy and help with that sort of thing. You can reference that option here: https://cassandra.apache.org/doc/latest/cassandra/operating/compaction/twcs.html

I would personally stay away from garbagecollect with TWCS tables, it tends to cause bigger problems with the time windows. If you are absolutely positive the window in question can be dropped, a better option is to bring the node down, manually delete the sstable set that should be expired, then bring the node online. If there is any question about the data, you can perform an sstabledump on the data file before dropping it.

Improve this answer
2
  • Thanks Paul. Issue resolved. I found the overlapped clustering key for 23rd Aug for these 3 nodes.Those were scattered for different dates and the latest sstable which holds the 23rd Aug Data,I could see the drop time would be 5.24 am IST on 17th October.I anticipated that at that time,all pending sstables across all 3 nodes will be dropped and thats exactly happened !!. I explained this to application team and ask them to try reprocess any data within same compaction window. It again proves how beautifully TWCS works (Unless we do mistakes) :)
    – Cassandra Thrift
    Commented Nov 2, 2023 at 10:14
  • Very nice, glad to hear you worked it out!
    – Paul
    Commented Nov 3, 2023 at 12:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.