1

Cluster of 3 cassandra nodes as a dev environment. Used to run in AWS on i3.2xlarge; migrated as a test to i3en.2xlarge to evaluate for later migrating production cluster to.

When running cassandra-stress benchmarks for large row counts (e.g. 100M and 1B), I am seeing an increase in write performance of 80-90%, but a decrease in read performance of around 80%. Minimal or no external traffic to the cluster during these tests.

I have checked the keyspace created by cassandra-stress for excessive SSTables, and there doesn't seem to be any problem there. Running iperf and I/O benchmarks on the new i3en nodes shows a 30-120% increase in performance over the i3 depending on the benchmark.

Improve this question
3
  • There could be a multiple reasons for this. For example, the compaction doesn't compact SSTables fast enough, and they pile up. Check how many pending compactions you have on nodes, and if there are many, increase compaction throughput - the default value (16Mbs) is very conservative. Also, for SSDs you can increase a number of concurrent compactors
    – Alex Ott
    Jul 10, 2019 at 14:36
  • Zero pending compactions; the keyspace used by cassandra-stress currently has 2 SSTables. The cluster has been performance tuned in the past, which I haven't touched much of since I inherited it but the i3en has very similar specs to the old node class, just newer CPUs and more and faster storage.
    – user5416
    Jul 10, 2019 at 14:40
  • Looking at the old benchmark results, it seems that the median latency has only gone up by 1ms, but the 99.9th percentile has gone from ~15-20 to ~60-70 which seems to be the only difference of any significance I can find and presumably is enough to slow the whole benchmark operation.
    – user5416
    Jul 10, 2019 at 14:47

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.