1

After we upgraded to Cassandra 3.11.15 from 3.11.14 we began to find that some small number of SSTables became corrupt when we ran cleanup. We have tried running a scrub on the corrupt SSTables but they are unrecoverable. By looking at the logs and monitoring the time stamps when the SSTables are created we have verified that the corruption is happening during cleanup. We do not see this happening during normal compaction or repairs.

The situation is that disks will fail on a host. We will take the host out of the ring to rebuild the disk. After the node is ready to go, we add it back in and run a full repair. Since the token ranges shift during this process, we run a cleanup after the repairs. Then later during normal repair and compaction, we discover two or three newly corrupted SSTables on different hosts. Is anyone else seeing this?

The SSTable Corruption error message is: "Failed to seek to next partition postion <large #>" The Exception is org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (): corruption detected, chunk at of length . at org.apache.cassandra.io.util.CompressedChunkReader$Map.readChunk(CompressedChunkReader:186)

We do not see any errors during the Cleanup but we do see a cleanup start for the table and we can see that the write time on the corrupt SSTable is soon after the Cleanup starts. Circumstantial evidence for sure but it has happened several times in a row.

1 Answer 1

1

Unfortunately, your procedure for dealing with a disk failure is incorrect. You should not be decommissioning a host if your intention is to put it back into service, particularly if you're just fixing the disk failure.

If you post a new question (separate to this one) with details of (1) the disk failure, and (2) the steps you are following to deal with it, I'd be happy to review and respond to it.

In any case, it's a completely separate issue from the corruption you originally posted. You haven't provided details of how you came to the conclusion that nodetool cleanup is leading to the corruption so I would suggest you update your original question above with the details and I'd be happy to review it.

Also, we would need the (3) full error message + (4) full stack trace to get an idea of what corruption you're facing. Cheers!

2
  • You are correct that the disk failure and the way we handle it is not directly related to the SSTable corruption. I put it in to provide background regarding why we run a cleanup from time to time. Given our situation, our process for handling failed disks works for us. Regarding my real question which is about the nodetool cleanup causing corruption, that has happened 4 times in a row. Unfortunately, I cannot post much info about the artifacts you request. This Cassandra instance is on a separate network and and I have to hand type errors here. I will type in the Caused-By above.
    – sastewa
    Aug 23, 2023 at 18:44
  • Without the (3) exact full error message + (4) exact full stack trace, I'm afraid I can't help you. And (5) you still haven't provided any details on how you came to the conclusion that cleanup is causing the corruption so I don't have much else to add. Cheers! Aug 24, 2023 at 1:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.