It is often repeated that the Big Data problem is that relational databases can not scale to process the massive volumes of data that are now being created.

But what are these scalability limitations that Big Data solutions like Hadoop are not bound to? Why can't Oracle RAC or MySQL sharding or MPP RDBMS like Teradata (etc.) achieve these feats?

I am interested in the technical limitations - I am aware that the financial costs of clustering RDBMS can be prohibitive.

  • 4
    Well, how those that claim that "relational databases can not scale to process the massive volumes of data that are now being created" justify their claim? – ypercubeᵀᴹ Feb 26 '12 at 9:00
  • I could rephrase to something simpler - what are the main scalability limitations of relational databases that stop them from growing out forever with more and more machines? – Jeremy Beard Feb 26 '12 at 11:50
  • 1
    @Jeremy: who says relational databases can't (won't) scale? Even Facebook is running on a relational database. See this article. – Marian Feb 26 '12 at 12:27
  • 1
    Only Vendors of non-relational kit say this? Note MySpace uses SQL Server – gbn Feb 27 '12 at 8:25
  • @Marian: Big Data isn't typically about the OLTP applications that generate data. Facebook themselves gave up on MySQL (and RDBMS altogether) nearly immediately for their data warehousing because it couldn't meet ETL SLAs. They use Hadoop to hold and crunch tens of petabytes of usage data. I'm curious what it was that meant they couldn't scale their RDBMS instead. – Jeremy Beard Feb 27 '12 at 11:36
up vote 15 down vote accepted

MS just had a tech talk in the Netherlands where they discussed some of this stuff. It starts off slowly, but gets into the meat of Hadoop around the 20 minute mark.

The gist of it is that "it depends". If you have a sensibly arranged, (at least somewhat) easy to partition set of data that (at least somewhat) is homogeneous, it should be fairly easy to scale to those high data volumes with an RDBMS, depending upon what you're doing.

Hadoop and MR seem to be more geared to situations where you are forced to to large distributed scans of data, especially when those data aren't necessarily as homogeneous or as structured as what we find in the RDBMS world.

What limitations are Big Data solutions not bound to? To me, the biggest limitation they're not bound to is having to make a rigid schema ahead of time. With Big Data solutions, you shove massive amounts of data into the "box" now, and add logic to your queries later to deal with the lack of homogeneity of the data. From a developer's perspective the tradeoff is ease of implementation and flexibility on the front end of the project, versus complexity in querying and less immediate data consistency.

  • Thanks Dave, you are getting me closer to what I am trying to find out. You say Hadoop is geared to situations with large distributed scans - if some/many RDBMS' have clustered solutions (RAC, shards, MPP, etc), why can't they also do that as well? What makes it infeasible for an RDBMS to sort 10 trillion records in 16 hours like a very large Hadoop cluster can? see here – Jeremy Beard Feb 27 '12 at 11:43
  • 2
    Nothing makes it infeasible for an RDBMS cluster to perform this kind of work, and you can configure an RDBMS to scale out to do this sort of thing. The problem with an RDBMS is that in order to do this, you have to be really careful about how you structure your schema and partitions in order for it to work. Big Data architectures win when your data aren't structured enough to be partitioned and optimized easily or effectively in an RDBMS. – Dave Markle Feb 27 '12 at 13:37
  • 1
    Incompetent db designers make it hard for relational databases to scale. Too many companies think applicatiopn developers can design databases (or worse use ORMS to do the design) when they need to hire comptent database developers from the start. The second person you hire for a project that involves data should be the database developer. – HLGEM Feb 27 '12 at 21:18
  • 3
    @HLGEM: My response to this is, "meh". The most effective developers are going to be the ones that understand both sides of the stack -- the idea that there's such a thing as a good "application developer" who works with an RDBMS all of the time without knowing how it works is a fallacy. Likewise, the idea that there's such a thing as a good "database developer" who doesn't understand ORMs or the application side of it is also, IMO, a fallacy. – Dave Markle Feb 28 '12 at 2:47

Database pioneer and researcher Michael Stonebraker co-wrote a paper that discusses the limitations of traditional database architectures. Generally, they scale up with more expensive hardware, but have difficulty scaling out with more commodity hardware in parallel, and are limited by legacy software architecture that was designed for an older era. He contends that the BigData era requires multiple new database architectures that take advantage of modern infrastructure and optimize for a particular workload. Examples of this are the C-store project, which led to the commercial database Vertica Systems, and the H-store project that led to VoltDB, an in-memory OLTP SQL database designed for high velocity BigData workloads. (Full disclosure, I work for VoltDB).

You might find this webinar interesting on this topic. It responds to some of the myths that have arisen with the success of NoSQL databases. Basically, he contends that SQL was not the problem, it shouldn't be necessary to give up traditional database features such as consistency in order to get performance.

It is not entirely true that RDBMS cannot scale. However, the partial truth in the statement depends on the architecture. In the list that you gave, Oracle RAC is different from the rest (sharded MySQL and Teradata). The major difference is shared disk vs shared nothing architectures.

Shared disk architectures like Oracle RAC suffer from scaling because at some point or other all the machines running should synchronize on some part of the data. For e.g. global lock manger is a killer. You can keep fine tuning it to some extent but you will ultimately hit a wall. If you cannot easily add machines, you should have fewer but super powerful machines which may burn your pocket. In case of shared nothing architectures(or sharded data), each machines takes ownership of some data. It need not synchronize with other mahcines if it want to update some data.

Then comes the breed of NoSQL databases. I would treat them a subset of traditional RDBMS databases. Not all applications in this world will need all the functionality offered by RDBMS. If I want to use database as a cache, I would not care about durability. May be in some cases I would also not care about consistency. If all my data lookup is based on a key, I dont need support for range queries. I may not need secondary indexes. I dont need the whole query processing/query optimization layer which all the traditional databases have.

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.