1

I have cluster cassandra-2.2.19-1 with 3 nodes. On each of them i have errors in system.log

ERROR [CompactionExecutor:2430] 2023-05-12 11:52:44,047 CassandraDaemon.java:231 - Exception in thread Thread[CompactionExe
cutor:2430,1,main]
java.lang.RuntimeException: Not enough space for compaction, estimated sstables = 1, expected write size = 140702127076
        at org.apache.cassandra.db.compaction.CompactionTask.checkAvailableDiskSpace(CompactionTask.java:299) ~[apache-cass
andra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:119) ~[apache-cassandra-2.2.19
.jar:2.2.19]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:74) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:257) ~[apache-cassandra-2.2.19.jar:2.2.19]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_332]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_332]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_332]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_332]
        at java.lang.Thread.run(Thread.java:750) [na:1.8.0_332]
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,963 CompactionTask.java:83 - insufficient space to compact all requested files. 577960.75MB required, BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-2409-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-29-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-1581-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-14-big-Data.db')
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:301 - Not enough space for compaction, 577960.75MB estimated.  Reducing scope.
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:83 - insufficient space to compact all requested files. 424087.53MB required, BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-2409-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-29-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-14-big-Data.db')
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:301 - Not enough space for compaction, 424087.53MB estimated.  Reducing scope.
WARN  [CompactionExecutor:2430] 2023-05-12 11:53:37,965 CompactionTask.java:83 - insufficient space to compact all requested files. 275999.3MB required, BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-2409-big-Data.db'), BigTableReader(path='/var/lib/cassandra/data/jcr_ss4c/ss4c-188b8f301c0e11e984b197f8b0cd7933/lb-29-big-Data.db')

I added a new node, but this did not solve the problem, and I see that the data in the cluster does not match in size

Datacenter: datacenter1
=======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address          Load       Tokens       Owns (effective)  Host ID                               Rack
UN  192.168.168.2 1003.24 GB  256          71.1%             f0e9a44f-2ae2-45c7-aa8c-6a36a4fc3333  rack1
UN  192.168.168.3  991.04 GB  256          77.4%             f11f037e-d581-4121-a6db-b2cc86a95555  rack1
UN  192.168.168.4  991.04 GB  256          73.9%             5add51ce-0ff5-45b6-9dac-c794962f4444  rack1
UN  192.168.168.5  775.13 GB  256          77.6%             8ca72d79-b747-43da-a7f7-3f3c66a92222  rack1

Can I run nodetool repair on nodes if there is a problem with compaction and there is not enough free space on the nodes. Will repair create temporary tables, which will negatively impact my disk space?

And is it possible in my situation to run a nodetool cleanup to free up space from old data?

Or, besides repair and cleanup, are there other solutions to the problem with compaction in my case?

Improve this question

1 Answer 1

Reset to default
1

Running nodetool cleanup on the three original nodes is definitely the correct procedure to evict partitions the nodes no longer own to free up space.

However if you haven't been running repairs regularly AND the nodes have been dropping mutations, there is huge risk of data loss since replicas will be inconsistent.

Our recommendation is to repair all the nodes in the cluster before performing a cleanup operation to prevent data loss. Cheers!

Improve this answer
4
  • I did not quite understand, your recommendations to do repair, despite the fact that there will be a risk of data loss? But will repair create temporary tables, which will worsen the situation with disk space?
    – Александр Хрисанов
    May 12 at 7:47
  • 1
    Repair will make replicas consistent preventing the data loss from a cleanup. Repair does NOT and NEVER has created temporary tables. I'm not sure where you got that idea from. 🙂 Cheers!
    – Erick Ramirez
    May 12 at 10:07
  • I ran the command repair --full and three days later I see this in the log Repair command #1 finished in 20 hours 38 minutes 3 seconds Starting repair command #2, repairing keyspace jcr_log with repair options... But more than a day has passed and nothing happens. Is it possible to somehow understand the process is working, or is it frozen?
    – Александр Хрисанов
    May 15 at 3:25
  • It's a completely different question to the one we're addressing in this post so please log a new question. Cheers!
    – Erick Ramirez
    May 15 at 4:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.