3

I am a newbie to Db's on a whole, and I came across a tutorial, wherein we needed to store ~700 Tb of data over a few years. It was mentioned that scaling SQL for such limits of data is not the best approach, and hence in the tutorial they went ahead with NoSQL. I have a few doubts regarding this:

  1. Is it possible to store such Tb's in a single machine, as per my understanding, horizontal scaling is not possible in SQL. Please correct this if I am wrong.
  2. What will be the latency comparison of SQL vs NoSQL. Because of horizontal scaling, will a noSQL db like DynamoDB give lower latency than a SQL Db.
5
  • 1
    Horizontal scaling isn't exclusive to NoSQL (even though they like to pretend it is). What's important is whether the data is the sort you want to store in a RDBMS. I'm not sure a tutorial that starts with "we're going to store 700TB of data" is the best if you're completely new to databases. Remember that there are tutorials and there are "tutorials". Don't trust everything you read.
    – Kayaman
    Commented Jun 15, 2018 at 8:19
  • So, in practice, the amount of data that we can store in NoSQL can also be stored in a RDBMS.
    – Sanchay
    Commented Jun 15, 2018 at 8:25
  • 1
    Microsoft seemed to achieve 1100TB back in 2008. That was before the NoSQL craze and people claiming that SQL is dead and everything is "web scale" and so forth.
    – Kayaman
    Commented Jun 15, 2018 at 8:30
  • Ok, thanks for the link! Finally, I wanted to understand latency impacts. Is there any good link that I can read to understand how latency is impacted in such cases.
    – Sanchay
    Commented Jun 15, 2018 at 8:31
  • That depends on a lot of things. You have a lot to learn before you need to worry about latency. Concentrate on finding good, reliable articles. Can you link the tutorial you're following?
    – Kayaman
    Commented Jun 15, 2018 at 8:38

3 Answers 3

3

Is it possible to store such Tb's in a single machine

Definitely. Though that machine becomes a large single-point-of-failure.

Building a petabyte scale storage system that is efficient and fault tolerant can itself be a challenge, doing it with cost efficiency too even more so. I assume that being a "newbie as DBs" also means you are not particularly experienced with storage solutions either, so you don't want to be trying to put together a custom build.

A not un-common arrangement is to have a couple of clustered machines running your chosen DBMS (SQL Server etc.) using an "off the shelf" SAN based storage system. I did a quick calc with the first search result for "petabyte SAN" and a ~700Tb arrangement came in at around £40,000 and that is just the storage unit and drives to populate it: add onto that racks to hold it all, decent spec machines for the clustered database servers, the enterprise licensing that is likely to be needed, ... This level of storage is never going to be cheap. You also need to factor in the ongoing costs: electricity to power it all and the required air conditioning too, replacing drives when they fail, and also the man-power to monitor and maintain it all!

You would almost certainly be better off looking at managed solutions, perhaps with one of the bigger cloud providers, and let them worry about much of the scaling, redundancy, reliability, power, etc. problems for you. For this scale you are unlikely to find an "off the shelf" price so you'll need to talk to them directly. Be warned: this amount of storage, particularly for rapid access, with good resiliency (in terms of both data security and service availability) is not going to be remotely cheap, there is no way it can be.

as per my understanding, horizontal scaling is not possible in SQL. Please correct this if I am wrong.

It is possible, though whether it is practical compared to other options depends on your data and access patterns.

What will be the latency comparison of SQL vs NoSQL.

Because of horizontal scaling, will a noSQL db like DynamoDB give lower latency than a SQL Db.

All of the above can only really be answered with "it depends" without a lot more information about the data your are needing to store, and at least a vague idea of expected retrieval patterns and other requirements.

Just knowing the size and expected growth of the data is not nearly enough for a more detailed discussion.

1

While the question has a tag of dynamodb, yes you can store as much as you need. It has the limit of 400K/item and 80K WCU and RCU on N.Virginia region which can be lifted up by the support. There is also a limit for local secondary index, which can be up to 10gb in size and there is no limit for global secondary indexes.

On the other hand question itself is more about SQL and dynamodb is not SQL. If you need like a SQL then you should check

https://aws.amazon.com/redshift/

0

Is it possible to store such Tb's in a single machine

Sure, if you have the storage capacity. Of course that's not something you'd do regardless of the tech being used. Database clusters have existed for decades, and smart people worked on them. NoSQL is a recent invention and you sometimes hear claims that make NoSQL seem "better" than SQL, even though you're comparing apples to oranges.

What will be the latency comparison of SQL vs NoSQL

That depends on what you're doing. Different workloads can give you different results. Depending on the data and usage, either one could be the wrong tech of choice, but it does not depend on the amount of data alone.

NoSQL is not "improved SQL". It's a complementary technology.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.