I need to do some experiments in HBase and Cassandra and to do that I need an adequate dataset.

The dataset I'm looking for has to be large enough (i.e. more than 2GB) and the data in it has to be sufficiently unstructured to be representative of the kind of problems that relational technology can't cope. Maybe data derived from social networks, and so on.

Does anyone have that kind of dataset or knows where can I find such a dataset?

closed as off-topic by Shawn Melton, Mark Sinkinson, Colin 't Hart, RolandoMySQLDBA, Paul White Jun 30 '15 at 15:33

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Shopping list question - questions about which tool, library, product or resource you should use are off-topic here because they quickly become obsolete and often are just about the preferences of the answerer. If you have an issue with or a question about a specific tool, please revise your question to conform to that scope." – Shawn Melton, Mark Sinkinson, Colin 't Hart, RolandoMySQLDBA, Paul White
If this question can be reworded to fit the rules in the help center, please edit the question.


[EDIT - added several public dataset sites].

First off, there is no real evidence that NoSQL databases are "better" at handling large datasets than traditional (OldSQL) RDBMSs. Check out Ted Dziuba's article about how he can't wait for NoSQL to die. He makes the point that Walmart continue to use RDBMSs - and they're not a small company! He says that NoSQL is, and should remain, niche and that in all likelihood, you don't need it. He also makes the reasonable point that Facebook, Google and Twitter are not normal companies with normal data processing needs.

Google Michael Stonebraker's writings on OldSQL, NoSQL and NewSQL (e.g. 1, 2, 3). He makes the point that NoSQL throws the baby out with the bathwater - i.e. NoSQL doesn't enforce ACID transactions and that this is an abomination for a database system. As you'll see from his bio, he's been involved in databases as an academic and in industry for 40 years.

He agrees with the NoSQL school that OldSQL (think Oracle, MS SQL Server &c.) is "old technology" and needs to be "sent to the home for retired software", and that OldSQL (in this case MySQL) has trapped Facebook in a "fate worse than death". His point about NewSQL is that for OLTP apps, you need a shared-nothing sharded architecture (check out his VoltDB) and that for OLAP you need dedicated columnar stores, i.e. Vertica (which he sold to HP).

If this doesn't convince you, check out Brian Aker's (former MySQL chief architect) humourous take on NoSQL here.

As for large datasets, I would urge you to Google in the area which particularly interests you. I know that metereological (my dad was one) datasets can be very large and also genomic datasets can also be huge (I studied genetics in uni). This site seems to be right up your alley - with many multi-GB and multi-TB datasets.

[EDIT] Other sites of interest are to be found (1, 2, 3, 4 & 5).

I would strongly urge you to benchmark both RDBMS and NoSQL solutions. As I mentioned, Dziuba says NoSQL is niche - it may suit your particular needs, I don't know. 2GB datasets are now officially small (even tiny - they fit easily on all memory sticks. Nowadays, you've got to be getting into the multiple terabyte region for a database to be getting large. Consider the Apollo moon landing's IT capacity. There was time when 2GB was huge - no so anymore!

Finally, I leave the last word to Ted Dziuba:

"I'm not just singling out Cassandra - by replacing MySQL or Postgres with a different, new data store, you have traded a well-enumerated list of limitations and warts for a newer, poorly understood list of limitations and warts, and that is a huge business risk".

  • 3
    -1 While I don't think you're wrong, I'd argue that an RDBMS and NoSQL data store simply do different things, making this a pretty unfair comparison. Beating up on an NoSQL data store because it's a bad RDBMS is as silly as beating up on an RDBMS because it's a bad file system. And ultimately, you didn't really do anything to answer the question actually being asked. – Bacon Bits Jun 30 '15 at 14:45
  • 1
    You obviously missed the 6 odd sites with large public datasets that I posted. I have studied stats, but it took me a while to dig them up. I am criticising NoSQL for abandoning ACID, not for being a "bad RDBMS". I believe the question has been more than answered. – Vérace Jun 30 '15 at 15:12
  • Thanks for datasets that you posted, I will analyze them. It's a unfortunate not find any dataset related with social networks. Regarding the comparison between RDBMS and NoSQL databases I think both are very useful and should be used according the environment and particular needs concerned – Pedro Cunha Jun 30 '15 at 15:51
  • Graph databases might be one of the niches where NoSQL do better. I did Google "large facebook datasets" and some of the results I posted talked about networks, Tweets and the like. I didn't look at them in much detail, but if you check, I'm sure that you'll find something relevant. Try "Large Twitter, Google, Amazon". There was definitely stuff about the American road system - different states - can be seen as nodes and edges in a graph. – Vérace Jun 30 '15 at 15:57
  • There's also the StackExchange open data forum and the datascience one. Your question might be on-topic there - read the charters. – Vérace Jun 30 '15 at 16:30

Not the answer you're looking for? Browse other questions tagged or ask your own question.