I have a large vector space model, in which each point is a multi-variate observation. In other words, I have a large number of pretty large records, mostly of ints and floats. I started with in-memory matrices, but given a large number of points (1B+) and a large number of dimensions (1K+), I soon realized that that wasn't an option.

I need to

1) make the whole thing persistent and dynamic (I want to add/update/remove points/dimensions easily)

2) query the VSM efficiently, performing similarity queries (given a point, find top K similar points) and range queries (find all points within a subspace).

Based on some reading, it seems that there isn't an off-the-shelf package to do this. What would you use to implement this kind of scenario? Dumping each point as a row in a relational db didn't work because the large number of dimensions. Even after normalization, the relational db was clearly too slow to perform similarity queries.

Moreover, I need to compute the dot product on chunks of the matrix. What technology would you recommend in this case? For example, I have read about array databases, and OLAP cubes, but I would like to have some advice from real DBAs.

Thanks, Mulone

closed as off-topic by Max Vernon, Mark Storey-Smith, Paul White, RLF, RolandoMySQLDBA Aug 30 '14 at 2:42

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Shopping list question - questions about which tool, library, product or resource you should use are off-topic here because they quickly become obsolete and often are just about the preferences of the answerer. If you have an issue with or a question about a specific tool, please revise your question to conform to that scope." – Max Vernon, Mark Storey-Smith, RLF
If this question can be reworded to fit the rules in the help center, please edit the question.


You should take a look at SciDB - a database designed for precisely this use case. It has as as its major driver Michael Stonebraker, a man who knows a lot about databases - it would be an exaggeration to say that he's been involved with more database systems than most people have had hot dinners, but not by much :-)!

"including Ingres, Illustra, Cohera, StreamBase Systems, Vertica, VoltDB, and Paradigm4. He was previously the Chief Technical Officer (CTO) of Informix. He is also an editor for the book Readings in Database Systems".

Google "michael stonebraker scidb sparse matrix" and subsets thereof and I think that it might be a good fit for what you want.

Not the answer you're looking for? Browse other questions tagged or ask your own question.