I'm going to receive a large group of CSV files daily. Additionally I have 1 billion records of data (from those CSV files) but it's not a fixed number (1b records daily). They're going to be growing and I need to store them in a DB. Also there are some extra points:
- There's no update
- There's no join and relationship
- Select a bunch of rows and group by
- Write intensive is 9 times more than read (we don't need to read data it must be a storage of data)
- I don't need normalisation
I had a bench between MySQL InnoDB and MyISAM. MyISAM was more better than InnoDB (because I have no normalisation) but MySQL is not a good approach because I have no relation.
Also I've checked MongoDB finally it took 50GB of data and used 150Gb of storage disk!
So I think I need a NoSQL DB which can do distributed write and support above extra points. What do you prefer?
I know I can use CSV files but I need a db approach not a file approach. I have a lot of tools and advantages like version update, bug fixes, security stuff, read and write performance, replica, etc in a database. I cannot have something like group by (or where, ...) in CSV files and I have no time to implement them with CSV files.