Corrupt empty row found in unfiltered partition

Question

I have a Cassandra cluster consisting of 15 nodes. The Cassandra version is 3.7.0

I am constantly getting an error message in the log messages of almost all nodes.

The recurring error message is exactly as follows:

ERROR [BatchlogTasks:1] 2023-03-22 10:13:20,281 CassandraDaemon.java:217 - Exception in thread Thread[BatchlogTasks:1,5,main]
java.io.IOError: java.io.IOException: Corrupt empty row found in unfiltered partition
    at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:224) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:212) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize30(PartitionUpdate.java:681) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.db.partitions.PartitionUpdate$PartitionUpdateSerializer.deserialize(PartitionUpdate.java:656) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:341) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.db.Mutation$MutationSerializer.deserialize(Mutation.java:350) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.batchlog.BatchlogManager$ReplayingBatch.addMutations(BatchlogManager.java:353) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.batchlog.BatchlogManager$ReplayingBatch.<init>(BatchlogManager.java:304) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.batchlog.BatchlogManager.processBatchlogEntries(BatchlogManager.java:237) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.batchlog.BatchlogManager.replayFailedBatches(BatchlogManager.java:207) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_66]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_66]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_66]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [na:1.8.0_66]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_66]
    at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
Caused by: java.io.IOException: Corrupt empty row found in unfiltered partition
    at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:388) ~[apache-cassandra-3.7.0.jar:3.7.0]
    at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer$1.computeNext(UnfilteredRowIteratorSerializer.java:219) ~[apache-cassandra-3.7.0.jar:3.7.0]
    ... 18 common frames omitted

I use the default configuration settings in cassandra.yaml.

I think the reason for this error is because of a table. Before this error, I was getting CorruptSSTableException error and I solved it with nodetool scrub.

Here's what I've tried to resolve this error:

• Restarting the nodes
• Running Nodetool scrub
• Running a repair with the Reaper tool

Answer 1

The stack trace indicates that the exception is thrown while replaying failed batches.

Looking at the code for UnfilteredSerializer.deserialize(), Cassandra runs into a problem deserialising a range tombstone and running into an empty row unexpectedly. In my experience, this is due to data corruption.

You mentioned that you see CorruptSSTableException errors. You need to figure out if there are problems with the data disk that is leading to the corruption because you will keep running into problems until you can address the root cause.

For a bit of background, [logged] batch mutations are stored in the system.batchlog table on two random nodes so if the data is corrupted on one node, Cassandra should still be able to replay it from the batch log on another node.

Note that by design, the system keyspace is configured with LocalStrategy so it is not replicated to other nodes which means that repairing the batchlog table will not help here.

As a side note, there are known issues with range tombstones and running a very old version of Cassandra is not ideal. This particular issue you are running into isn't very common and although I'm not saying upgrading to the latest version of the 3.11 line will fix it, it doesn't help that C* 3.7 is 6 years old (released in June 2016) and there are countless amount of fixes for known issues released since. Cheers!