1

Background for this question

  1. I have a machine learning pipeline where all our Features and pre-calculated prediction scores are stored in PostgreSQL database.
  2. I have in excess of 800 columns as features for my Machine Learning models - so having them in PGSQL is probably not a good idea because a. SQL is probably not ideal to have such a wide list of columns - i.e. columnar data b. Feature Data may be better stored in distributed database

Typical Use Case

  1. I need to read the features (not immediately) after I have written them - in batches - so that I can run them through my ML models and calculate scores (and store them in Database)
  2. Typically, I am talking about reading a batch of 100k records at a time, but it might be smaller - however, it might involve reading many, many columns (as I said above)

Why I wanted to look into Cassandra

  1. AP database - from CAP
  2. Column-based
  3. Good for fast writing ( may be reading too?)
  4. Updated My data volume is growing faster than I thought - already nearing 0.5 TB.

What I read online

  1. Cassandra is very good for writing data fast
  2. Reading from Cassandra is not as efficient as write

So what are your question(s) ?

  1. Why is Cassandra so good in writing, if I cannot read what I've written fast enough?
  2. What areas should I check in Cassandra, before giving it a try for my ML Feature Store for models?

My apologies if this question is too general, or too specific. Happy to review it again based on feedback.

Thanks in advance,

1
  • And technically, Cassandra is a "partitioned rowstore," not a columnstore.
    – Aaron
    Sep 26 at 15:40

1 Answer 1

1

You are misinformed. Cassandra is designed for high-velocity reads at internet scale.

This statement is incorrect:

Reading from Cassandra is not as efficient as write

Quite the opposite. If you have modelled your data correctly for Cassandra, the tables are optimised for super-fast reads.

Reading 100K records is not a lot for production Cassandra clusters I've worked on. It is realistic to scale a Cassandra cluster for 1M reads/sec or even 10M reads/sec.

Cassandra scales linearly. If you have a 3-node cluster that is capable of handling 50K reads/sec and you need to do 100K reads/sec, you just double the size of your cluster to 6 nodes. If the traffic to your app doubles to 200K reads/sec, then all you need to do is add another 6 nodes for a total of 12 nodes.

When done correctly, you can keep scaling Cassandra to meet your throughput requirements. Cheers!

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.