I am planning a for a database that i expect will be very big in terms of number of records rather than tables and batabases. My Question is what product (i.e, SQL 2008, Cassandra, Azure) handles scaling better?

I expect that there will be 100k or so records will be added everyday, It should easily retrieve a single row and a batch of them.

The app will crawl the web and for each word in a dictionary will find the same word used in different sentences and record them to a database. The developers are mainly .net oriented but they are fluent in C/C++ as well and because this is a academic project anything is basically affordable.

One platform/product per answer please.

  • Define "gigantic" and "platform" please. Jan 5, 2011 at 14:16
  • 2
    1) How much data - what seems "very big" to you might be small to someone else 2) what access method will you used, e.g. will you mainly be inserting or updating? will you be accessing individual rows or large ranges in SELECTs?
    – Gaius
    Jan 5, 2011 at 14:24
  • 3
    While I think SQLite and MSAccess will not work for you, databases like MySQL, PostgreSQL, Oracle *** and DB2 are proven to do the thing you want. Be more specific. What features do you need? Who will use the 100k records per day? Which systems do you need to connect? What programming languages do your developers speak? Can you do PL/SQL? What budget you have.. Jan 5, 2011 at 14:37
  • Depending on what you're doing, I've also seen people break things up by year, and then use a UNION if they're gong to be searching across year boundriess. This is particularly useful for databases like mySQL where you don't have as fine control of how to spread your data out across spindles, or where you have (had?) limitations on table size (either number of records or total size on disk)
    – Joe
    Jan 5, 2011 at 15:16
  • 1
    @Sevki "Has anyone tried Denali CTP" is a different question. Please try to phrase it as such on a new question, but make it a "correct" question please. dba.stackexchange.com/questions/how-to-ask
    – jcolebrand
    Jan 6, 2011 at 2:40

3 Answers 3


Any of the big RDBMS systems or NoSQL systems can handle that sort of data load. Back in 2002 I had a database table with 1B records in it that got more than 100k records a day, and that was running SQL Server 2000. The systems today are much faster (both the hardware and the software) and should be able to handle that load going in with no problem. The big problem becomes getting the data out, and having the data indexed properly to find it based on how you want to search. And having enough memory to load the data into cache to find it faster.


I don't know which is the best, but I was running some tests on 3 tables, using Guid, BigInt and Int as primary keys.

Each table have 1,000,000 entries. To retrieve a whole table it takes 12 seconds on my machine. To retrieve 80,000 it takes about a second.

SQL Server is surely very reliable.

There is this page on microsoft.com that has some stuff about performance and scalability, maybe you can find something else if you need.

  • A table with a guid as a primary key will become very fragmented, very quickly as data is put in, unless the guids are sequential.
    – mrdenny
    Jan 5, 2011 at 21:40
  • @mrdenny So wouldn't you use a heap table and have a index to account for that?
    – jcolebrand
    Jan 6, 2011 at 2:41

You could consider GridSQL, which can be placed on top of PostgreSQL to enable intraquery parallelization. It is GPL.

Not the answer you're looking for? Browse other questions tagged or ask your own question.