I have a huge sparse matrix ( 10^9 rows, 10^6 cols, density < 0.03% ), where each row has at least one nonzero column, but some column may contain only zeros. The cells are decimal numbers > 0.
I am looking for some db (preferably key-value) that can retrieve whole row or whole column as fast as possible. Also, I don't need to do any analytics in the DB.
I've come across SciDB which should be fast with multidimensional data, but I am afraid it is too complex for my needs.
Also, another option is to use SQL db ( probably Postgres ) but that is a bit slow and can't be scaled as easily as most NoSQL ( I expect fast row increase in matrix ).
So my biggest hope is some key-value storage, but I am not sure how to represent the matrix.
- Using CRS format - but I have no idea how to implement it using key-value store.
- Maybe something like { key, key, value } and have indexed both keys, but I am not sure if this is even possible. - So far I've found some clues about secondary index, but I can't imagine how would it look like.
I have some experience with SQL databases but almost none with NoSQL :(