2

we have 40 node Cassandra 4.0.7 cluster in GCP, after single node failure the entire cluster goes bonkers on CPU usage and I/O waits. Tried to bump the cluster size to 50 nodes the same is happening but a bit slower. Cluster stabilizes immediately after the down node transition into UN state.

Nodes have 16 vcpu 32G RAM, 2TB data ssd ( ~700G used ), commit logs are on spinning os partition. The Java heap is 16G. The cluster has 1 namespace with 1 table and 2 materialized view with replication factor 3 and TWCS.

nodetool tpstats do not show anything blocked , but during the problem the MutationStage jobs started to pile up. I tried to move commit log to SSD on a single node, I saw less I/O but CPU consumption and Pending mutations were still happening.

Any hints where to look?

I can provide more information ( in details ) if needed.

snippet from logs: ( abc.actions is the table name not the view name)

ERROR [MutationStage-8] 2023-03-23 15:29:38,933 Keyspace.java:652 - Unknown exception caught while attempting to update MaterializedView! abc.actions
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
    at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicas(ReplicaPlans.java:170)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicasForWrite(ReplicaPlans.java:113)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:354)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:345)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:339)
    at org.apache.cassandra.service.StorageProxy.wrapViewBatchResponseHandler(StorageProxy.java:1312)
    at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:1004)
    at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:167)
    at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:647)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:513)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:494)
    at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:147)
    at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
    at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
    at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
ERROR [MutationStage-7] 2023-03-23 15:29:38,932 Keyspace.java:652 - Unknown exception caught while attempting to update MaterializedView! abc.actions
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
    at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicas(ReplicaPlans.java:170)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicasForWrite(ReplicaPlans.java:113)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:354)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:345)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:339)
    at org.apache.cassandra.service.StorageProxy.wrapViewBatchResponseHandler(StorageProxy.java:1312)
    at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:1004)
    at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:167)
    at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:647)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:513)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:494)
    at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:147)
    at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
    at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
    at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
ERROR [MutationStage-8] 2023-03-23 15:29:38,933 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-8,5,main]
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
    at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicas(ReplicaPlans.java:170)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicasForWrite(ReplicaPlans.java:113)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:354)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:345)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:339)
    at org.apache.cassandra.service.StorageProxy.wrapViewBatchResponseHandler(StorageProxy.java:1312)
    at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:1004)
    at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:167)
    at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:647)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:513)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:494)
    at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:147)
    at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
    at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
    at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)
ERROR [MutationStage-7] 2023-03-23 15:29:38,933 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[MutationStage-7,5,main]
org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level ONE
    at org.apache.cassandra.exceptions.UnavailableException.create(UnavailableException.java:37)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicas(ReplicaPlans.java:170)
    at org.apache.cassandra.locator.ReplicaPlans.assureSufficientLiveReplicasForWrite(ReplicaPlans.java:113)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:354)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:345)
    at org.apache.cassandra.locator.ReplicaPlans.forWrite(ReplicaPlans.java:339)
    at org.apache.cassandra.service.StorageProxy.wrapViewBatchResponseHandler(StorageProxy.java:1312)
    at org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:1004)
    at org.apache.cassandra.db.view.TableViews.pushViewReplicaUpdates(TableViews.java:167)
    at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:647)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:513)
    at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:494)
    at org.apache.cassandra.service.paxos.PaxosState.commit(PaxosState.java:147)
    at org.apache.cassandra.service.paxos.CommitVerbHandler.doVerb(CommitVerbHandler.java:34)
    at org.apache.cassandra.net.InboundSink.lambda$new$0(InboundSink.java:78)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:97)
    at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:45)
    at org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165)
    at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137)
    at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
    at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
    at java.base/java.lang.Thread.run(Thread.java:834)

UPDATE: Attaching an image with data drive resource utilization accorss the cluster. When I do benchmarking tests for the drives I always get at least 10X more throughput before hitting any I/O.

Is it possible that some setting are not set properly set for my hardware?

For example I have:

concurrent_reads: 32
concurrent_writes: 32
concurrent_counter_writes: 32
concurrent_materialized_view_writes: 32

enter image description here

2 Answers 2

1

The symptoms you described definitely indicate that the nodes are IO-bound and cannot keep up with the writes.

It isn't so much a case of a single node taking down the whole cluster -- it's just that that node is the first one to go down and all other nodes are soon to follow anyway.

Using spinning HDDs for commitlog is not recommended because it is not fast enough to handle high write loads. For what it's worth, mutations (in MutationStage) are Cassandra writes.

I realise cost is a major factor but you should consider switching to SSDs, particularly NVMe SSDs if you can afford them. Otherwise, you will need to keep scaling out by adding more nodes to increase the capacity of your cluster. Cheers!

2
  • Thank you @Erick. When my cluster does not have issues read and write latencies are very small ( according to cfstats writes ~ 0.030 ms , reads ~ 15 ms ), resource consumption ( CPU , I/O ) is ok as well. Only "bad" thing is of heap memory mapping ( I have 32G RAM ) , mmap show that Cassandra process needs to map 700G. I can change to map only index files. So my question is based on what I should start scaling my cluster? Thanks again
    – vahagn
    Commented Mar 27, 2023 at 18:48
  • Your problem is not latency -- it's write throughput at peak traffic. 🙂 Commented Mar 27, 2023 at 21:02
0

I meet the problems. The root cause is that the table has more than one materialized view. maybe is a bug.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.