Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I am planning a for a database that i expect will be very big in terms of number of records rather than tables and batabases. My Question is what product (i.e, SQL 2008, Cassandra, Azure) handles scaling better?

I expect that there will be 100k or so records will be added everyday, It should easily retrieve a single row and a batch of them.

The app will crawl the web and for each word in a dictionary will find the same word used in different sentences and record them to a database. The developers are mainly .net oriented but they are fluent in C/C++ as well and because this is a academic project anything is basically affordable.

One platform/product per answer please.

share|improve this question
Define "gigantic" and "platform" please. –  Peter Eisentraut Jan 5 '11 at 14:16
1) How much data - what seems "very big" to you might be small to someone else 2) what access method will you used, e.g. will you mainly be inserting or updating? will you be accessing individual rows or large ranges in SELECTs? –  Gaius Jan 5 '11 at 14:24
While I think SQLite and MSAccess will not work for you, databases like MySQL, PostgreSQL, Oracle *** and DB2 are proven to do the thing you want. Be more specific. What features do you need? Who will use the 100k records per day? Which systems do you need to connect? What programming languages do your developers speak? Can you do PL/SQL? What budget you have.. –  Sebastian Roth Jan 5 '11 at 14:37
Depending on what you're doing, I've also seen people break things up by year, and then use a UNION if they're gong to be searching across year boundriess. This is particularly useful for databases like mySQL where you don't have as fine control of how to spread your data out across spindles, or where you have (had?) limitations on table size (either number of records or total size on disk) –  Joe Jan 5 '11 at 15:16
@Sevki "Has anyone tried Denali CTP" is a different question. Please try to phrase it as such on a new question, but make it a "correct" question please. dba.stackexchange.com/questions/how-to-ask –  jcolebrand Jan 6 '11 at 2:40
show 2 more comments

closed as not constructive by Sathya, jcolebrand, Brian Ballsun-Stanton, Coder Hawk, Joe Jan 6 '11 at 19:07

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.If this question can be reworded to fit the rules in the help center, please edit the question.

3 Answers

Any of the big RDBMS systems or NoSQL systems can handle that sort of data load. Back in 2002 I had a database table with 1B records in it that got more than 100k records a day, and that was running SQL Server 2000. The systems today are much faster (both the hardware and the software) and should be able to handle that load going in with no problem. The big problem becomes getting the data out, and having the data indexed properly to find it based on how you want to search. And having enough memory to load the data into cache to find it faster.

share|improve this answer
add comment

I don't know which is the best, but I was running some tests on 3 tables, using Guid, BigInt and Int as primary keys.

Each table have 1,000,000 entries. To retrieve a whole table it takes 12 seconds on my machine. To retrieve 80,000 it takes about a second.

SQL Server is surely very reliable.

There is this page on microsoft.com that has some stuff about performance and scalability, maybe you can find something else if you need.

share|improve this answer
A table with a guid as a primary key will become very fragmented, very quickly as data is put in, unless the guids are sequential. –  mrdenny Jan 5 '11 at 21:40
@mrdenny So wouldn't you use a heap table and have a index to account for that? –  jcolebrand Jan 6 '11 at 2:41
add comment

You could consider GridSQL, which can be placed on top of PostgreSQL to enable intraquery parallelization. It is GPL.

share|improve this answer
add comment

Not the answer you're looking for? Browse other questions tagged or ask your own question.