1

I have cluster cassandra-2.2.19-1 with 3 nodes. On each of them i have errors in system.log

ERROR [CompactionExecutor:2430] 2023-05-12 11:52:44,047 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExe
cutor:2430,1,main]
java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 140702127076
        at org.apache.cassandra.db.compaction.CompactionTask.checkAvailableDiskSpace(CompactionTask.java:299) ~[apache-cass
andra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:119) ~[apache-cassandra-2.2.19
.jar:2.2.19]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:257) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_332]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_332]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_332]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_332]
        at java.lang.Thread.run(Thread.java:750) [na:1.8.0_332]
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,963 CompactionTask.java:83 - insufficient space to compact all requested files. 577960.75MB required, BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-2409-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-29-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-1581-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-14-big-Data.db')
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:301 - Not enough space for compaction, 577960.75MB estimated.  Reducing scope.
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:83 - insufficient space to compact all requested files. 424087.53MB required, BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-2409-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-29-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-14-big-Data.db')
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:301 - Not enough space for compaction, 424087.53MB estimated.  Reducing scope.
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:83 - insufficient space to compact all requested files. 275999.3MB required, BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-2409-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-29-big-Data.db')

I added a new node, but this did not solve the problem, and I see that the data in the cluster does not match in size

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
UN  192.168.168.2 1003.24 GB  256          71.1%             f0e9a44f-2ae2-45c7-aa8c-6a36a4fc3333  rack1
UN  192.168.168.3  991.04 GB  256          77.4%             f11f037e-d581-4121-a6db-b2cc86a95555  rack1
UN  192.168.168.4  991.04 GB  256          73.9%             5add51ce-0ff5-45b6-9dac-c794962f4444  rack1
UN  192.168.168.5  775.13 GB  256          77.6%             8ca72d79-b747-43da-a7f7-3f3c66a92222  rack1

Can I run nodetool repair on nodes if there is a problem with compaction and there is not enough free space on the nodes. Will repair create temporary tables, which will negatively impact my disk space?

And is it possible in my situation to run a nodetool cleanup to free up space from old data?

Or, besides repair and cleanup, are there other solutions to the problem with compaction in my case?

1 Answer 1

1

Running nodetool cleanup on the three original nodes is definitely the correct procedure to evict partitions the nodes no longer own to free up space.

However if you haven't been running repairs regularly AND the nodes have been dropping mutations, there is huge risk of data loss since replicas will be inconsistent.

Our recommendation is to repair all the nodes in the cluster before performing a cleanup operation to prevent data loss. Cheers!

4
  • I did not quite understand, your recommendations to do repair, despite the fact that there will be a risk of data loss? But will repair create temporary tables, which will worsen the situation with disk space? Commented May 12, 2023 at 7:47
  • 1
    Repair will make replicas consistent preventing the data loss from a cleanup. Repair does NOT and NEVER has created temporary tables. I'm not sure where you got that idea from. 🙂 Cheers! Commented May 12, 2023 at 10:07
  • I ran the command repair --full and three days later I see this in the log Repair command #1 finished in 20 hours 38 minutes 3 seconds Starting repair command #2, repairing keyspace jcr_log with repair options... But more than a day has passed and nothing happens. Is it possible to somehow understand the process is working, or is it frozen? Commented May 15, 2023 at 3:25
  • It's a completely different question to the one we're addressing in this post so please log a new question. Cheers! Commented May 15, 2023 at 4:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.