2

Background for this question

  1. I have a machine learning pipeline where all our Features and pre-calculated prediction scores are stored in PostgreSQL database.
  2. I have in excess of 800 columns as features for my Machine Learning models - so having them in PGSQL is probably not a good idea because a. SQL is probably not ideal to have such a wide list of columns - i.e. columnar data b. Feature Data may be better stored in distributed database

Typical Use Case

  1. I need to read the features (not immediately) after I have written them - in batches - so that I can run them through my ML models and calculate scores (and store them in Database)
  2. Typically, I am talking about reading a batch of 100k records at a time, but it might be smaller - however, it might involve reading many, many columns (as I said above)

Why I wanted to look into Cassandra

  1. AP database - from CAP
  2. Column-based
  3. Good for fast writing ( may be reading too?)
  4. Updated My data volume is growing faster than I thought - already nearing 0.5 TB.

What I read online

  1. Cassandra is very good for writing data fast
  2. Reading from Cassandra is not as efficient as write

So what are your question(s) ?

  1. Why is Cassandra so good in writing, if I cannot read what I've written fast enough?
  2. What areas should I check in Cassandra, before giving it a try for my ML Feature Store for models?

My apologies if this question is too general, or too specific. Happy to review it again based on feedback.

Thanks in advance,

1
  • And technically, Cassandra is a "partitioned rowstore," not a columnstore.
    – Aaron
    Commented Sep 26, 2022 at 15:40

1 Answer 1

1

You are misinformed. Cassandra is designed for high-velocity reads at internet scale.

This statement is incorrect:

Reading from Cassandra is not as efficient as write

Quite the opposite. If you have modelled your data correctly for Cassandra, the tables are optimised for super-fast reads.

Reading 100K records is not a lot for production Cassandra clusters I've worked on. It is realistic to scale a Cassandra cluster for 1M reads/sec or even 10M reads/sec.

Cassandra scales linearly. If you have a 3-node cluster that is capable of handling 50K reads/sec and you need to do 100K reads/sec, you just double the size of your cluster to 6 nodes. If the traffic to your app doubles to 200K reads/sec, then all you need to do is add another 6 nodes for a total of 12 nodes.

When done correctly, you can keep scaling Cassandra to meet your throughput requirements. Cheers!

2
  • Can there be situations where cassandra does reads 3 times faster than writes? Or is it a sign that the cluster is not configured properly? I have a 5 node cluster and my reads are 60k/s but writes are 18k/s. My data is a table with a bigint (as primary key) and 2 lists (100 elements in each list, in each row). I don't have clustering keys and my reads just retrieve rows by the bigint id column, without any filtering. Commented Apr 22 at 23:46
  • This is different to what was asked above so please consider posting a new question otherwise the comments section will end up being a never-ending thread. Cheers! Commented Apr 23 at 6:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.