1

We are running STCS compaction nodewise. Until last month, it was taking around 3 days per node. This month took 11 days per node. No change or significant data insert/delete happened. What are the driving factors for improving Compaction time? Want to review and revisit so as to improve timing.

1 Answer 1

1

What are the driving factors for improving Compaction time?

So SizeTieredCompactionStrategy is really just the combination of like-sized files. That process is largely bottlenecked by how fast your drives are.

By default, compaction is throttled so that it doesn't take too many resources away from the node's actual, regular functions. You can control this with the compaction_throughput setting (MiB/s) in the yaml. To see its current setting, either check the yaml or run:

nodetool getcompactionthroughput

To change it, again you can modify the yaml and restart, or use:

nodetool setcompactionthroughput <value>

If you want your compactions to run twice as fast as they normally do, check that value and then double it. One thing that I've done before, is to have a cron job which did a nodetool setcompactionthroughput 256 at night (during slow hours) and then set it back to 8 during business hours.

Till last month it was taking around 3 days per node, this month took 11 days per node.

So diagnosing this can be tricky. Like I said, compaction is largely bottlenecked by how fast your drives are. Bigger files will also take longer, too. Check the size of the data files for the table in question. If those file sizes have grown into the hundred GB range, you might want to use sstablesplit on them.

Otherwise, check that something else isn't monopolizing the drives during that time. At my last job, we used to build OpenStack VMs with drives aggregated behind a series of NetApp devices. When things suddenly got inexplicably slower, we asked the storage team to investigate. Turns out that a new cluster was seeing high write activity, and many of its drives were provisioned in the same storage array as the cluster in question, basically stealing its IOPS.

Not saying that's your exact issue, but you might want to make sure nothing else is using those drives.

the splitup sstables will be the candidate of immediate TWCS cycle? or it will be unaffected as originated days back?

Yes, if TWCS is in use for that table, I would think those files would be candidates for compaction right away.

7
  • Thanks a lot Aaron. We will try this. Commented Jan 24, 2023 at 5:42
  • 1
    Hi Aaron,Can we change the value on the Fly...say compaction is running and we are changing the value of this parameter ...is this possible and effect immediately? Commented Jan 24, 2023 at 9:20
  • @CassandraThrift Yes, if you change it using nodetool setcompactionthroughput it is effective immediately.
    – Aaron
    Commented Jan 24, 2023 at 12:31
  • Thanks Aaron. Actually I am in a new system and facing different issues. Here 2 different TTLs are being used for same table data which is resulting huge number of SStable pileup as TWCS is not able to remove sstable effectively as TTL eipiration time is different.Thus major compaction being done.I could see 2 big SStable files of 9 GB and 226 gb whereas usual file size is 150-250 MB. In case we do sstable split....after we start cassandra,the splitup sstables will be the candidate of immediate TWCS cycle? or it will be unaffected as originated days back ? Commented Jan 25, 2023 at 6:32
  • Also will it be possible that some sstables will be generated with TTL expired data entirely and dropped immediately so some storage would be cleared Commented Jan 25, 2023 at 6:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.