We are using Cassandra version 3.11.5 and team was using "nodetool compact" command for compaction which had created very big sstable of 250 GB. This time we have ran "nodetool compact -s" expecting it will split the files to smaller chuncks however this never happened and still we have one big sstable file generated post compaction. Am I missing any steps or config change here. We never got extended downtime approval for running sstablesplit
-
See here why forcing a major compaction is not a great idea.– MadhavanCommented Mar 9, 2023 at 16:54
-
Hey Madhavan. Absolutely agree with you. However I have got this system with all these issues for malpractice since years.I am looking into resolve this issue to have a well tuned database. I am in process of asking for downtime of node one at a time (5 nodes Cluster) so as to apply sstablesplit . Before that I tried "nodetool compact -s"– Cassandra ThriftCommented Mar 10, 2023 at 8:05
-
Apache Cassandra 4.1.x is already out there and I guess you would love the guardrails feature to apply guardrails to prevent anti-patterns from happening from the applications interacting with this cluster. See cassandra.apache.org/_/blog/… for more details.– MadhavanCommented Mar 10, 2023 at 19:31
Add a comment
|
1 Answer
The "split" option (-s or --split-output) in the nodetool compact command generates SSTables which are split into multiple files which are 50% then 25% then 12.5% of the original table size until the smallest chunk is 50MB in size.
For example if the total size of a table on a node is 100GB, the split option will generate SSTables with the following sizes:
- 50GB (50% of total)
- 25GB (25% of total)
- 12.5GB (12.5% of total)
- 6.25GB (6.25% of total)
- and so on
with the smallest SSTables are 50MB in size.
If this isn't the behaviour you are seeing on the node, you will need to update your original question and provide additional details including the investigation you've done, table statistics and minimal steps to replicate the problem.Cheers!
answered Apr 17, 2023 at 6:16
-
Hi Erick,Let me give you some more detail. In this project team configured TWCS compaction with compaction window 1 minute whereas file insertion frequency is 20-30 minute.As a result no TWCS in effect (nodetool compactionhistory shows no entry) ,then team started manual compaction with nodetool compact command which brings different time data into the big ssatble created with this major compaction,I could see last 14 months data and not being dropped neither further compacted.My first choice was sstablesplit however downtime not permitted so tried nodetool compact with -s . Continued.... Commented Apr 18, 2023 at 6:53
-
I still can see the file size even increased.I expected this big one would also be split,which never happened.Now I am planning to change compaction window as 2 days so bucket number would be < 50 as TTL is 90 days. Then I will take downtime node wise,run sstablesplit on that Big sstable to split into 50 mb files.Can I expect here that some split sstables where all data expired would be dropped automatically? The remaining split files with fully active or overlapped (active and expired) would take part in TWCS compaction as soon as I start cassandra ? Continued.... Commented Apr 18, 2023 at 7:02
-
Even if thats the case,I believe all these old data would be removed max after 90 days. If those split sstables with older data won't take part in twcs,then time to time ,those would be expired and automatically dropped? Kindly help me to correct my understanding. We will never run major compaction again Commented Apr 18, 2023 at 7:04