Cassandra log reports "MarshalException: UUID should be 16 or 0 bytes (14)"

Question

While monitoring the logs of Cassandra I'm getting the below error in the logs. It's not causing any issue till now, just want to confirm if it's a potential issue & how can I rectify this error.

INFO  [IndexSummaryManager:1] 2022-08-24 03:28:57,657 IndexSummaryRedistribution.java:78 - Redistributing index summaries
ERROR [ReadStage-1] 2022-08-24 03:30:56,390 AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread Thread[ReadStage-1,5,main]
java.lang.RuntimeException: org.apache.cassandra.serializers.MarshalException: UUID should be 16 or 0 bytes (14)
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_272]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134) [apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:113) [apache-cassandra-3.11.8.jar:3.11.8]
    at java.lang.Thread.run(Thread.java:748) [na:1.8.0_272]
Caused by: org.apache.cassandra.serializers.MarshalException: UUID should be 16 or 0 bytes (14)
    at org.apache.cassandra.serializers.UUIDSerializer.validate(UUIDSerializer.java:43) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:132) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.dht.AbstractBounds.format(AbstractBounds.java:130) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.dht.AbstractBounds.getString(AbstractBounds.java:123) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.db.PartitionRangeReadCommand.queryStorage(PartitionRangeReadCommand.java:286) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:430) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1884) ~[apache-cassandra-3.11.8.jar:3.11.8]
    at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2652) ~[apache-cassandra-3.11.8.jar:3.11.8]
    ... 5 common frames omitted
INFO  [CompactionExecutor:38035] 2022-08-24 04:28:35,665 AutoSavingCache.java:395 - Saved CounterCache (339956 items) in 375 ms

Answer 1

The exception was thrown during a read request as indicated by the ReadStage thread in the error message:

ERROR [ReadStage-1] 2022-08-24 03:30:56,390 AbstractLocalAwareExecutorService.java:166 - \
  Uncaught exception on thread Thread[ReadStage-1,5,main]

As part of reading the data in the SSTable, it was validating a UUID column and expecting there to be either 16 or 0 bytes but it got (14). The most likely issue is that the data in the table is corrupted.

The log entries won't tell you which table is problematic or which partition has bad data in it so unless you can identify it from the application logs, there isn't much you can do about it. Cheers!

[EDIT] I should've mentioned that if the issue really is corrupted data, you can attempt to rebuild the SSTables with nodetool scrub.