Disk space is not released after compaction

Question

We have deleted some of the partionons of some tables in cassandra database. But after the delete space was not recliamed becuase of size-tiered compaction . Manualy compacted tables with -s options. we have cluster of having 2 data ceneters and each with 6 nodes, we ran compaction both the DC's of all nodes. But Disk usage on DC1 reduced but 6% increase on the other DC2. So please let me know the reason for this and also suggest me on the next steps 2 bring down the usage of disk on Dc2. for your information even load on both DC's are come down.

Even cleared old snapshots of around 1.3TB on all nodes

o/p command as of below comment 4 (find /path/to/dir/ -printf '%s %p\n'| sort -nr | head -10):

12434906818 /data1/cassandra/data/witsml/log_curve-c9311570d40d11e7af9a6b6facb54bee/.maxdatetimeindex_idx/md-283562-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-08-05/md-680-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-07-29/md-680-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-07-22/md-680-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-07-15/md-680-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-07-08/md-680-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-07-01/md-680-big-Data.db
4111835201 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-06-24/md-680-big-Data.db
4111835201 /data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/md-680-big-Data.db
3782418777 /data1/Full_Backup/data1/cassandra/data/system_distributed/parent_repair_history-deabd734b99d3b9c92e5fd92eb5abf14/snapshots/2022-08-05/md-1963-big-Data.db

Answer 1

So for possible reasons why space isn't reclaimed after running a major compaction, I can think of the following:

• A snapshot of a table is hard-linked to the SSTable file(s).
• The tombstones' gc_grace_seconds for the table has not expired (tombstones are still valid and won't get collected).

I would double-check the gc_grace_seconds (default of 10 days...864000 seconds) and maybe try running it again.

NOTE: Future Googlers, avoid running nodetool compact at all costs.

Compaction will likely never happen organically ever again, which means you'll need an ongoing ops process to run it regularly.