Lets assume that I'm creating a social network like instagram or facebook with petabytes of data and my needs are:
1: High availability
2: Data mining and NLP (basically anything related with AI)
3: lots of reports and aggregations
4: Search and relations ofc

and my question is:
1: Which databases do I need to use
2: How to store data to optimize it for search, data mining and...

I don't care about redundency or resources I just want to have near real time results.

  • This would be an opinion-based question/answer; IE: All the Oracle fans might suggest Oracle, and MySQL fans might suggest MySQL. And the scope of the question would require huge amount of research by experts with far more list of requirements. – Mark Stewart Jul 6 at 16:52
  • 1
    Eh? "don't care about redundency" versus "1. High availability"?? – Rick James Jul 6 at 18:33
  • @MarkStewart I just tagged mysql and oracle just because they have lots of followers, I don't want to use only 1 DB I'm looking for an answer to let me know for instance if you want to save likes you may need to use cassandra db or for that purpose use mongo or use elasticsearch, as I said I'm looking for a design and solution and not to know which DB is the best one... – AH.Pooladvand Jul 6 at 19:30

Realistically, the right answer varies depending on datasize of "megabytes" versus "gigabytes" versus "terabytes". "Petabytes" would literally take a ton of disk drives. And thousands of servers. And lots of load balancers, routers, etc. Once you add HA, double everything (at least). What's your budget like? Do you have someone on staff full time to replace the disk drives as they fail?

If you want to tone down the "petabytes" and focus on one of the other requests at a time, I will be happy to provide more details.

| improve this answer | |
  • Currently we have like 100TB of data on elasticsearch and one mistake that we did as rookies was to store data on ES like RDBMS, we want to redesign the whole system and convert/tweak our data but we're not sure how to design it – AH.Pooladvand Jul 6 at 19:45
  • Let's assume that we have users, posts, likes, comments like instagram and lets assume that a user has a 500 posts each post has 1M likes and 1M comments, so the question is how should I store in a nosql database what kind of databases do I need, how do I store that data in nosql db, how to store it in RDBMS database, is it a good idea to store it in both rdbs and nosql db? and how should I manage my data, what kind of tools do I need and etc – AH.Pooladvand Jul 6 at 19:51
  • I mean how to store my data to have the best performance to get a user followers, likes or to get all comments of a post and also to be able to get very fast aggregations and reports and how to store my historical data, and also considering high IO on my data since data scientists are running jobs, manipulating data and so on... – AH.Pooladvand Jul 6 at 19:57
  • @AH.Pooladvand - Do you a "count" of likes? Or a list of "who" likes whom? See my discussion of Summary tables: mysql.rjweb.org/doc.php/summarytables – Rick James Jul 6 at 21:14
  • 1
    @AH.Pooladvand - RDBMS vs NoSQL vs ES: Count the disk hits. Seriously, when the technology does not provide anything other than a full scan of 100TB, performance sucks. MySQL + Summary tables provides a much more scaling. ES provides techniques for certain types of search, but I don't know how well it scales. Is your data already shared? – Rick James Jul 6 at 21:21

Not the answer you're looking for? Browse other questions tagged or ask your own question.