Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

We have come to a point where for a few of our customers' systems, at peak times, events are generated at that crazy rate. That is, millions of new events per minute (between 1M and 2M).

The records/docs size are 1.5Kb on average.

Is there a database that can handle such radical write speed out there (16K - 32k inserts per second)?

There are several candidates out there, Cassandra, Hive, Hadoop, HBase, Riak and perhaps a few others, but I would like to know from someone else's experience and not just quoting from each database system's website self testimonials.

If you have handled such load at your work, your advice is highly appreciated.

share|improve this question

closed as too broad by Mark Storey-Smith, RolandoMySQLDBA, Justin Cave, ConcernedOfTunbridgeWells, Phil Aug 16 '13 at 9:29

There are either too many possible answers, or good answers would be too long for this format. Please add details to narrow the answer set or to isolate an issue that can be answered in a few paragraphs.If this question can be reworded to fit the rules in the help center, please edit the question.

Think about how banks handle billions of consistent transactions. Through sharding and batch processing you can handle any number of transactions, it is just a case of using the right structure rather than throwing hardware at the wrong structure. – JamesRyan Aug 16 '13 at 11:04

Ok, let me start simple: My old trusted 5 year old main db server (2x quad core AMD Opteron 2378) handles around 80,000 inserts per second, though small ones. That is without SSD for log and data on a SQL Server.

There are plenty of databases that can meet your requirement - it is MOSTLY a question of appropriate hardware. Heck, if you would go Oracle, then the smallest Exadata in a performance configuration (pure SSD) likely can do 10 times as much - we got in a project 5 gigabyte/second stable IO out of it, with a paltry 3 storage nodes.

That said, you WILL have to test it yourself. A lot depends on your programming or lack thereof. If you hit 1 insert per transaction, that is VERY different - and a lot harder to do (not at all because your log partitions need to handle 5 million IOPS then or fake the flush).

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.