Sometimes Cassandra nodes go down because there is not enough free disk space

Question

I need some advices for tuning Cassandra

(I think it's mb data modeling problems (large partitions), I wanna try to tune it with settings)

We faced a problem with disk space, it's sometimes crashing a node. If I'm right it happens when node starting some major compaction (up to 8 TB (from nodetool compactionstats)) and doing it slow, while we have some writes and/or another big compaction (n TB).

• Cassandra 3.11.7
• 64 cores (we have a really low utilization, i don't understand how to increase it)
• 8 nodes and 2 dc (4 nodes per dc)

Settings for one of the biggest table:

• compaction = {'class': 'TimeWindowCompactionStrategy', 'compaction_window_size': '2', 'compaction_window_unit': 'DAYS', 'max_threshold': '32', 'min_threshold': '4', 'tombstone_threshold': '0.5'}
• compression = {'chunk_length_in_kb': '64', 'class': 'LZ4Compressor'}
• replication = {'class': 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes = true;
• default_time_to_live = 5184000
• gc_grace_seconds = 86400
• if you need another info i can add it

I've tried change some compaction settings:

• concurrent_compactors 4 -> 2 -> 3 (I thought that if I did this, then the risk of large compactions in parallel would be lower)
• compaction_throughput 32 -> 256 -> 512 -> 1024 -> 2048 (it looks like it has some limitations or smth, cause I'm increasing this option, and from some value major compactions are not significantly accelerating)
• tombstone_threshold: default -> 0.5

Answer 1

The TimeWindow compaction strategy compacts the current window as STCS, and once the end of the time window has been reached, the final compaction for that time window is performed and the resulting files are then part of the historic aspect of the time window and will not be re-compacted again, except when the file is being entirely removed.

You have a TTL of 60 days, and a day of GC Grace Seconds, so the TWCS files are eligible for removal after 61 days.

Since it is only the current window which is compacting, a number of the changes you made would not have had a significant impact - with a caveat below.

concurrent_compactors : it's impacting the current window on the TWCS table only, and will impact any other STCS / LCS tables.

compaction_throughput : Same as the concurrent_compactors and you are correct that just increasing it does not linearly increase compaction throughput. The value is shared between compactors, and giving 2 compactions 2048 mb/sec throughput threshold will not have them achieve that, compaction itself takes time and has a logical throughput limit.

caveat - these apply for normal compaction - if you manually kick off a major compaction on the TWCS table then both settings do apply since it is going to put everything into a single time window and its going to destroy the ability for the TWCS to operate properly.

You mention 8 TB being compacted - which sounds like a manually initiated compaction of all of the data - this is a bad thing. The number of sstables on your chart drops as well dramatically, so it sounds like the TWCS is being manually compacted into a single file.

3-5 TB per node - is achievable if its all under TWCS but has significant drawbacks, I would recommend considering a greater number of nodes and scale horizontally, and not vertically. It is recommended to run more nodes, with less CPU and achieve a greater scale / throughput in comparison to fewer nodes with more cpu. I would be also looking to increase the RF if you can, using just 1/1 is less than ideal from a resiliency and consistency perspective.

Final item is the partition sizes - from the chart there is a partition size issue, you should be aiming for smaller partitions, 100 MB is often used, although Cassandra 4 can often mention 1 GB as the soft max - these though are both maximum values and not optimal - please do not consider them as targets. If you can keep the partition size below 10Mb, you will have less struggles.