1

After upgrading a 6 nodes Cassandra cluster from 3.11 to 4.1 the read performance degraded and IO usage is higher. Also during repairs the IO usage becomes very high ceiling on the aws node limit of 3000 read IOps. The write performance is slightly better or the same but we had few milisec writes anyway so not a that visible.

Cassandra 4 runs on:

  • latest AmazonLinux 2023, kernel 6.1,
  • R6i.2xlarge instance with 8 cpu/64G RAM
  • single EBS gp3 vol for data, 3000 iops.
  • Java 11, Shenandoah GC with 31GB heap

Cassandra 3.11 was running on:

  • AmazonLinux 2, kernel 4.14
  • R5.2xlarge instance with 8 cpu/64G RAM
  • Java 8, CMS GC with 16GB heap.
  • single EBS gp3 vol for data, 3000 iops.

Cassandra settings remained mostly the same from version 3.11, however we moved from java8 with CMS GC and 16G heap to java 11/Shenandoah and 31G heap.

The high IO usage was not present in 3.11.

Anybody else experienced higher IO on reads and degradation of performance (higher read latency) after migration to Cassandra 4? Any key setting or functionality that changed between versions and could impact reads?

Thank you

2 Answers 2

1

I haven't heard of reports of degraded performance in Cassandra 4.1.

Assuming this is in your production system, I would analyse the traffic to verify if the reads have increased compared to when the cluster was running with Cassandra 3.11 to validate that you're comparing apples for apples.

In my experience, high disk IO is caused by high application traffic resulting in more read requests. Cassandra on its own doesn't cause high IO -- it won't read from disks unnecessarily unless the application requests a read.

Also, I noted that you said "Cassandra settings remained mostly the same from version 3.11". You need to qualify this statement because you shouldn't be using C* 3.11 configuration because it has changed significantly in C* 4.x.

You should start with a vanilla C* 4.1 configuration files and adjust the settings based on your testing. Don't just copy the settings from your original C* 3.11 cluster. Cheers!

1
  • Hi, yes reads remained the same. Indeed we started with the default cassandra 4 config and adjusted some values that we had also changed on the cassandra 3. But as I wrote in my answer it looks like the smaller amount of offheap memory is the main candidate for this performance degradation. Maybe worth mentioning that our datasize or Load - how nodetool reports it, is 1.6 TB per node.
    – Catalin M
    Commented Jun 8, 2023 at 8:37
0

I'll add this answer as a follow-up on the findings.

After some investigation we found out that the larger Heap was the main culprit for the performance degradation. In Cassandra 3.11 we had CMS GC with 16G Heap on same 64G RAM size, but with the upgrade we increased Heap to 31G from 64G RAM. Also this was breaking the default recommendation for a 1/4 of RAM for Heap.

That left less memory available for OS file caches and I guess this was the reason for increased IO and performance degradation. Reducing the Heap made a big difference as the reads returned to acceptable values.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.