0

Cluster background: We have a three node cassandra cluster. As soon as I clear the entire database and start from scratch soon the load will reach about 100 GB per node and keep on increasing and after a few weeks it stabilises somewhere around 300-400 GB per node. We have SASI indexes enabled so the space occupied is on the higher side. This database stores all the mointoring data from my cluster and hence we have a configured TTL of 48 hours.

Problem: I have a complex cqlsh query to plot certain graphs based on the response of these queries; when the load is low [let's say around 80 to 100 GB per node] the queries work just fine and I get desired results but as soon as the load reaches 250+ GB per node we start getting totally empty responses for all these same cqlsh queries. One observation is that these queries never timeout; I always get the responses in less than 5 seconds but they are just empty rows, no errors no warnings and no exceptions. Incase it was a hardware issue like the CPU is not enough or storage speed is not upto the mark then I think we should get a timeout rather then empty responses

Workaround: If I issue a major compaction on the keyspaces created by my application the load reduces from 300 GB to less than 100GB per node and then again it starts working as expected. However, from the various articles on the internet I understood that running major compactions is not recommended and also the compactions are taking very very long time to finish so its not very ideal for us. I checked the compactionstats and the minor compactions are executed as expected but they are just not sufficient. Also we tried tweaking the min/max thresholds of the keyspace but now luck. The compaction strategy is STCS for my tables

Any suggestions on what is going wrong here??

1 Answer 1

0

Without the schema and the query, it's really impossible for anyone else to know what is going on. But my best guess is that you're running a full table scan which would explain the behaviour you described.

Cassandra is designed for super-fast single-partition retrieval, with partitions distributed across nodes in a cluster. It is not designed for full table scans which require retrieving partitions from all nodes in the cluster.

Run a trace on the queries so you have a better idea of where it's slowing down and there's a good chance that it will also tell you why.

In any case if you need to perform analytics on your data, consider using Spark since it optimises the underlying CQL queries by breaking them up into segments and distributes the load across replicas. Cheers!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.