Scalable DBaaS that can scale to hundreds of terabytes with easy backup and restore

Question

I've gone over many DBaaS solutions, including Xeround (limited to 50G, in memory), NuoDB (currently in beta 7), DrawnToScale (not out yet), MongoDB (in memory), RavenDB (for ASP.NET, but NoSQL) and many others, DynamoDB (costly for real-time web app and doesn't offer backup/restore).

I find everyone lacking in some ways. I search for a scalable and affordable Database as as service solution using NewSQL (RDMBS), but I will ocnsider NoSQL if there aren't any solutions for my needs.

Amazon DynamoDB seems like a good solution because I can scale to terabytes of data without worrying about disk space. However, DynamoDB costs a lot for high read / write applications and it doesn't offer easy backup/restore option in case of data corruption that is very important to me.

In short what I need:

1) DBaaS (can be self hosted too) the can easily scale to terabytes of data (ie. big data)

2) Capable of very high throughput for real-time application that does many read/writes (although I will use caching too to reduce overload)

3) Being able to query the data using SQL, a RDBMS DB. Although if a NewSQL solution isn't available, I will go with NoSQL

4) Easy backup and restore big amount of data in case of data corruption. I know that many solutions are safe against server failure, but I worry about data corruption and I want to be able to restore the data back if that happens - IMPORTANT!

5) I will be developing my web app / mobile app in ASP.NET, so I prefer a solution that have a .NET connector

6) If hosted, I prefer hosting my DB on Amazon AWs.

7) I don't want a self-managed solution and handle sharding and all the other complicated stuff

8) I want an affordable solution, not like Clustrix that costs thousands of dollars per node. I want a cost-effective solution where I can start small and pay more when my database grows (all idea behind cloud architecture, right?)

Right now the only solution that looks ok is RavenDB, NuoDB (still in beta). Solution like Xeround is amazing, and I am using it right now, but it's in-memory DB and limited for 50GB, so it's not good for a bog data hosting.

It seems that new solutions appear every month and it seems like there isn't any good solution out there (a new startup maybe?). I wanted to know what are my options. As a developer, I really want to focus on building big applications, but I can't start until I have a reliable DB solution that can handle very high traffic and can scale in size without me needing to build all the DB server architecture myself.

I spent weeks trying to find a solution with no luck. A need a solution like Xeround but for big data. Who knows, maybe there is one up there that I missed. Please help me out.

Answer 1

Barry Morris here from NuoDB. Answers below:

1) DBaaS (can be self hosted too) the can easily scale to terabytes of data (ie. big data)

• NuoDB is a downloadable product that runs anywhere (laptop, rack, public cloud). We have not announced DBAAS at this point. NuoDB uses Key/Value stores at the storage layer so supported DB sizes will be related to whatever the particular KV store can handle.

2) Capable of very high throughput for real-time application that does many read/writes (although I will use caching too to reduce overload)

• NuoDB is a general purpose database system designed for high-speed scaled-out transaction processing. There is no need for a separate cache tier because the NuoDB in-memory tier performs a similar function.

3) Being able to query the data using SQL, a RDBMS DB. Although if a NewSQL solution isn't available, I will go with NoSQL

• Check. NuoDB is a SQL database.

4) Easy backup and restore big amount of data in case of data corruption. I know that many solutions are safe against server failure, but I worry about data corruption and I want to be able to restore the data back if that happens - IMPORTANT!

• NuoDB supports JDBC, ODBC and other standards APIs, so backups can be done at the SQL level. Other alternatives include simply taking one of your storage managers offline and putting it somewhere safe.

5) I will be developing my web app / mobile app in ASP.NET, so I prefer a solution that have a .NET connector

• NuoDB has an ODBC API and there is a community developed ASP.NET connector.

6) If hosted, I prefer hosting my DB on Amazon AWs.

• NuoDB supports Amazon AWS

7) I don't want a self-managed solution and handle sharding and all the other complicated stuff

• No sharding or partitioning (or caching, M/S replication etc) needed with NuoDB.

8) I want an affordable solution, not like Clustrix that costs thousands of dollars per node. I want a cost-effective solution where I can start small and pay more when my database grows (all idea behind cloud architecture, right?)

• NuoDB is free forever for a baseline system.

Answer 2

The only backends that I am aware of that can scale to hundreds of TB are Hadoop and Bigtable. There are absolutely no backends that support hundreds of TB that supports easy backup and restore. That's nearly an impossibility with that quantity of data.

For "a few" TB, you can use PostgreSQL, MySQL, SQL Server, DB2, MongoDB, CouchDB, or pretty much any relational or non-relational DB out there.

How much data and traffic do you actually have and how much do you expect to have within the next 1-2 years? Plan your requirements around that and don't optimize prematurely.