Cloned Cassandra cluster returns gossip IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1

Question

After cloning Cassandra database from one cluster to another cluster using nodetool refresh we are seeing below errors often. This might not be related to migration but not sure. What causes this issue? Is there any explanation? How to fix this and does this cause any issues?

ERROR 16:27:50 java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
    at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178)
    at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465)
    at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895)
    at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78)
    at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240)
    at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
    at java.lang.Thread.run(Thread.java:748)

java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
    at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) [apache-cassandra-3.11.5.jar:3.11.5]
    at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) [apache-cassandra-3.11.5.jar:3.11.5]
    at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) [apache-cassandra-3.11.5.jar:3.11.5]
    at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) [apache-cassandra-3.11.5.jar:3.11.5]
    at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) [apache-cassandra-3.11.5.jar:3.11.5]
    at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) [apache-cassandra-3.11.5.jar:3.11.5]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_222]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_222]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_222]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_222]
    at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
    at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_222]

Answer 1

The IllegalStateException is getting thrown when the Gossiper class attempted to evict a node from gossip, possibly because the node being evicted is invalid.

This leads me to suspect that there are gossip peers which are not part of the cluster. Is it possible that you cloned all the data including the system keyspaces?

Some system keyspaces and tables are local to a node and are only ever valid for the node where it is stored. If you copy those system tables to another node, they are going to be invalid as they will contain data which doesn't belong to the other node.

Only ever clone application data, never system data, as I've documented in How do I restore Cassandra snapshots to another cluster with identical configuration? since it will lead to unpredictable cluster behaviour. Cheers!