sstableloader leads to CompactionExecutor error, "IllegalArgumentException: Self-suppression not permitted" due to NullPointerException

Question

When I try to load the SStable in through sstableloader, I am getting this following error:

ERROR [CompactionExecutor:28] 2023-09-26 10:48:37,680 JVMStabilityInspector.java:68 - Exception in thread Thread[CompactionExecutor:28,5,CompactionExecutor]
java.lang.IllegalArgumentException: Self-suppression not permitted
        at java.base/java.lang.Throwable.addSuppressed(Throwable.java:1054)
        at org.apache.cassandra.utils.Throwables.merge(Throwables.java:66)
        at org.apache.cassandra.db.lifecycle.Helpers.prepareForObsoletion(Helpers.java:140)
        at org.apache.cassandra.db.lifecycle.LifecycleTransaction.doPrepare(LifecycleTransaction.java:206)
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
        at org.apache.cassandra.io.sstable.SSTableRewriter.doPrepare(SSTableRewriter.java:386)
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.doPrepare(CompactionAwareWriter.java:104)
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.prepareToCommit(Transactional.java:168)
        at org.apache.cassandra.utils.concurrent.Transactional$AbstractTransactional.finish(Transactional.java:179)
        at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.finish(CompactionAwareWriter.java:114)
        at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:224)
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
        at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
        at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
        at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:359)
        at org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:113)
        at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
        at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.lang.NullPointerException: null

Cassandra version is 4.1

I have also given right permission and also given heap space of 80G (256GB ram totally).

But when the loading is almost complete 99%, the sstable loader freeze and this error thrown in the system log file. Is it due to memory issue or something else?

Answer 1

I've never come across this problem before which leads me to believe that it is extremely rare. Judging from the error message and the stack trace, I can think of two possibilities:

1. it's hitting a race condition while attempting to read an SSTable, or
2. the SSTable is corrupted.

Given that it was reported by the JVMStabilityInspector class, I think it is more likely that it was a race condition. If your cluster is overloaded, it could be the trigger for it.

However, it is also possible that Helpers.prepareForObsoletion() got a SSTableReader that couldn't read the SSTable on disk because it is corrupted.

The error was thrown by the CompactionExecutor:28 thread. If you go backwards in the log, there should be a corresponding INFO message with the thread ID CompactionExecutor:28 that will tell you which file was being opened/read. Temporarily move that SSTable (and all its components -- *-Data.db, *-Index.db, *-Statistics.db, etc) to another location then try to run sstableloader again. If it succeeds then it will confirm that the failure was due to a suspect SSTable. Cheers!