We are running STCS compaction nodewise. Until last month, it was taking around 3 days per node. This month took 11 days per node. No change or significant data insert/delete happened. What are the driving factors for improving Compaction time? Want to review and revisit so as to improve timing.
What are the driving factors for improving Compaction time?
SizeTieredCompactionStrategy is really just the combination of like-sized files. That process is largely bottlenecked by how fast your drives are.
By default, compaction is throttled so that it doesn't take too many resources away from the node's actual, regular functions. You can control this with the
compaction_throughput setting (MiB/s) in the yaml. To see its current setting, either check the yaml or run:
To change it, again you can modify the yaml and restart, or use:
nodetool setcompactionthroughput <value>
If you want your compactions to run twice as fast as they normally do, check that value and then double it. One thing that I've done before, is to have a cron job which did a
nodetool setcompactionthroughput 256 at night (during slow hours) and then set it back to 8 during business hours.
Till last month it was taking around 3 days per node, this month took 11 days per node.
So diagnosing this can be tricky. Like I said, compaction is largely bottlenecked by how fast your drives are. Bigger files will also take longer, too. Check the size of the data files for the table in question. If those file sizes have grown into the hundred GB range, you might want to use sstablesplit on them.
Otherwise, check that something else isn't monopolizing the drives during that time. At my last job, we used to build OpenStack VMs with drives aggregated behind a series of NetApp devices. When things suddenly got inexplicably slower, we asked the storage team to investigate. Turns out that a new cluster was seeing high write activity, and many of its drives were provisioned in the same storage array as the cluster in question, basically stealing its IOPS.
Not saying that's your exact issue, but you might want to make sure nothing else is using those drives.
the splitup sstables will be the candidate of immediate TWCS cycle? or it will be unaffected as originated days back?
Yes, if TWCS is in use for that table, I would think those files would be candidates for compaction right away.