Background for this question
- I have a machine learning pipeline where all our Features and pre-calculated prediction scores are stored in PostgreSQL database.
- I have in excess of 800 columns as features for my Machine Learning models - so having them in PGSQL is probably not a good idea because a. SQL is probably not ideal to have such a wide list of columns - i.e. columnar data b. Feature Data may be better stored in distributed database
Typical Use Case
- I need to read the features (not immediately) after I have written them - in batches - so that I can run them through my ML models and calculate scores (and store them in Database)
- Typically, I am talking about reading a batch of 100k records at a time, but it might be smaller - however, it might involve reading many, many columns (as I said above)
Why I wanted to look into Cassandra
- AP database - from CAP
- Column-based
- Good for fast writing ( may be reading too?)
- Updated My data volume is growing faster than I thought - already nearing 0.5 TB.
What I read online
- Cassandra is very good for writing data fast
- Reading from Cassandra is not as efficient as write
So what are your question(s) ?
- Why is Cassandra so good in writing, if I cannot read what I've written fast enough?
- What areas should I check in Cassandra, before giving it a try for my ML Feature Store for models?
My apologies if this question is too general, or too specific. Happy to review it again based on feedback.
Thanks in advance,