Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have the feeling that most of the database systems originate in the '80s and stereotypically implement tables, ACID transactions and constraints. They were build having in mind the scarcity of memory, disk and processing power.

I am wondering if there is any storage system (not necessary accessible through SQL) that is able to handle the following:

  • graphs (querying a la SPARQL).
  • matrix (n-dimensional) also sparse. Support of trivial matrix algorithms such as SVD, clustering.
  • efficient management of large (terabyte size) of data that doesn't change continuously; the change being happening though daily batches.
  • make use of large disk systems (14TB of RAID5 is today less than $1500). This means, more space for indexes, precalculated results, etc.
  • make use of GPU/multiple cores/processors/nodes, for a large query and indexing.

I know most of the items are implemented somewhere (Apache Cassandra, SPARQL, Netezza, Exadata), but I don't have knowledge of any product that may implement all.

share|improve this question
add comment

1 Answer

up vote 4 down vote accepted

I think a lot of these items are very much on the horizon (or beyond).

SPARQL, for instance, is something that I don't see databases incorporating any time soon. The closest I've seen is SDB, an interface that processes SPARQL and sends it off to a standard database.

Also, using GPUs as generic processors is still kind of a revolutionary thing. It hasn't quite caught on in the database world. At this stage, it's still in the world of academia and theory.

There's only one group (that I could find) that is developing a database to take advantage of the GPU. Alenka is an open source project but it's still very much in development.

Also, there's a new sorting algorithm called GPUTeraSort out there on the horizon. But (being an algorithm), I don't know of any specific databases that use it at this point.

Finally, there's a site, GPGPU, for general purpose computing on GPUs that you might want to keep an eye on. As databases arise that use the GPU, this will be the site that reports it.

Having said all of that, using multiple cores or multiple processors is almost the status quo. SQL Server, MySQL, Oracle--all of the major databases use multi-threading.


Ultimately, the items that you are asking for are something that is currently far beyond the database world.

You might also try cross-posting this on StackOverflow, as they might have some ideas of how to deal with Terabyte-level data using graphs, SPARQL, GPU enhancements, etc. However, their answer is probably going to be something like, "Yes, you can do it, but it would be a huge custom-built system."

share|improve this answer
    
Thank you for the GPU analysis, I was not aware of Alenka. I am considering developing such a storage engine, the reason I asked was that I felt somehow alienated from the rest of the world. Indeed the major databases use multithreading, but they are not able to use more than one thread to build an index or answer a single query. Am I wrong? Also, I imagine combining graphs with tables in the same query, such as " select the user who bought a server and has a fof who bought a tablet " fof=friend of a friend, also with multiple loops. Bought items are in a table while relation is in a graph. –  Razvan Popovici Aug 23 '11 at 15:18
    
It's possible to use multiple threads for the same query, providing the query is sufficiently complex. I don't know of any databases that use multiple threads on the same aspect of a query. It would be possible to break apart a table scan, for example, but not possible to do that with a nest loop. Whether there are any databases that break apart queries into multiple threads is a different question. –  Richard Aug 23 '11 at 15:20
    
As far as the "graph", that sounds like something that could be done in SQL, although maybe not as easily as it could in other languages. (I couldn't say for sure.) –  Richard Aug 23 '11 at 15:21
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.