BatchlogManager repeatedly reports NullPointerException error on one of 48-node cluster

Question

We have 48 Nodes Cassandra Cluster and we are getting a repeated error in only one node.Any idea what may be the reason. Error is as below :

ERROR [BatchlogTasks:1] 2023-08-29 09:12:55,524  CassandraDaemon.java:139 - Exception in thread Thread[BatchlogTasks:1,5,main]
java.lang.NullPointerException: null
        at org.apache.cassandra.batchlog.BatchlogManager$ReplayingBatch.<init>(BatchlogManager.java:367)
        at org.apache.cassandra.batchlog.BatchlogManager.lambda$processBatchlogEntries$0(BatchlogManager.java:290)
        at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.onNext(Flow.java:1227)
        at org.apache.cassandra.utils.flow.Threads$SchedulingTransformer.lambda$onNext$0(Threads.java:326)
        at org.apache.cassandra.concurrent.TPCRunnable.run(TPCRunnable.java:101)
        at org.apache.cassandra.concurrent.IOScheduler$PooledTaskWorker.run(IOScheduler.java:319)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
        at java.lang.Thread.run(Thread.java:748)
        at org.apache.cassandra.utils.concurrent.InlinedThreadLocalThread.run(InlinedThreadLocalThread.java:251)
        at org.apache.cassandra.concurrent.IOThread.run(IOThread.java:46)
        Suppressed: org.apache.cassandra.utils.flow.Flow$FlowException: Flow call chain:
        PagerFlow
        SchedulingTransformerIO scheduler: cached workers: 7(IOScheduler.java:183)
        ReduceToSingle      org.apache.cassandra.batchlog.BatchlogManager$$Lambda$2231/914170156@7b2cb757(BatchlogManager.java:286)
                at org.apache.cassandra.utils.flow.Flow.wrapException(Flow.java:2683)
                at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.onError(Flow.java:1270)
                at org.apache.cassandra.utils.flow.Flow$ReduceSubscriber.onNext(Flow.java:1231)
                ... 13 common frames omitted

Answer 1

Before the mutations in CQL batches are written, they are first persisted (logged) to the system.batchlog table on two random nodes. When the batch write is completed successfully, the corresponding batchlog entry is deleted. If the batch write fails for whatever reason then the nodes will attempt to replay the batch from the log until the destination nodes receive them, at which point the batchlog entry is deleted.

Looking at the code that replays batches in the BatchlogManager class, it seems like it's failing when trying to get the batch mutation size. This leads me to suspect that there is either (a) a corrupt entry in the system.batchlog, or (b) one or more batchlog SSTable is corrupted.

I suggest reviewing the rest of the Cassandra log(s) for clues. Note that the system.batchlog is "local" only, meaning it is not replicated to other nodes. Cheers!