0

I'm going to receive a large group of CSV files daily. Additionally I have 1 billion records of data (from those CSV files) but it's not a fixed number (1b records daily). They're going to be growing and I need to store them in a DB. Also there are some extra points:

  1. There's no update
  2. There's no join and relationship
  3. Select a bunch of rows and group by
  4. Write intensive is 9 times more than read (we don't need to read data it must be a storage of data)
  5. I don't need normalisation

I had a bench between MySQL InnoDB and MyISAM. MyISAM was more better than InnoDB (because I have no normalisation) but MySQL is not a good approach because I have no relation.

Also I've checked MongoDB finally it took 50GB of data and used 150Gb of storage disk!

So I think I need a NoSQL DB which can do distributed write and support above extra points. What do you prefer?

I know I can use CSV files but I need a db approach not a file approach. I have a lot of tools and advantages like version update, bug fixes, security stuff, read and write performance, replica, etc in a database. I cannot have something like group by (or where, ...) in CSV files and I have no time to implement them with CSV files.

closed as off-topic by Paul White Dec 23 '16 at 13:33

If this question can be reworded to fit the rules in the help center, please edit the question.

2

It seems that you have some analytics data. So basically all you need is a columnar database. E.g.:

  • Open source (NoSQL): druid.io
  • Commercial on-site (SQL & NoSQL) : MemSQL, VoltDB, etc
  • SAAS: Amazon Redshift, Google BigQuery

They have append-only semantic, archive your content to save the space, group by query is available (in druid.io it's called timeseries query), and use CPU for queries and transparently scale up to petabytes of data.

2

I did something like this with MySql and you can save directly CSV files in MySQL. I think Apache Cassandra is a good choice for you

  • CQL3 is very similar SQL, but with some limitations that come from the scalability (most notably: no JOINs, no aggregate functions.)
  • CQL3 is now the official interface. Don't look at Thrift, unless you're working on a legacy app. This way, you can live without
    understanding ColumnFamilies, SuperColumns, etc.
  • Querying by key, or key range (secondary indices are also available)
  • Tunable trade-offs for distribution and replication (N, R, W)
  • Data can have expiration (set on INSERT).
  • Writes can be much faster than reads (when reads are disk-bound)
  • Map/reduce possible with Apache Hadoop
  • All nodes are similar, as opposed to Hadoop/HBase
  • Very good and reliable cross-datacenter replication
  • Distributed counter datatype.
  • You can write triggers in Java.

You see the difference between NoSQL databases here http://kkovacs.eu/cassandra-vs-mongodb-vs-couchdb-vs-redis

Not the answer you're looking for? Browse other questions tagged or ask your own question.