I'm currently testing out Apache Hbase for our Matrix analytics service. I'm using a managed cluster running on AWS EMR.

The matrices are sparse, and we have 50,000 columns, and up to 10 million rows. The values are integer values

The main operation we'd like to support is random access of any 500 rows and 30 columns. We would like this operation to return in under a second, and ideally under half a second.

Apache Hbase seemed like the ideal option, as it's advertised to support 'real-time' access of large sparse matrices with 'millions of columns and rows'.

The instance I'm running is sizable, with 4 nodes, each with 16 cores and 256GB of memory.

I've tried both 'wide and short' and 'tall and thin' table formats. The 'wide and short' format has 50,000 columns, and is simply the matrix represented in hbase. The 'tall and thin' format has the row key as a composite key, with the format '{row_name};{column_name}' - e.g. F1S4_160106_001_B0111;100507661. The 'tall and thin' format has a single column containing the value.

For both formats, I've reduced the block size to 8192 bytes and turned on the Bloom Filter, as well as adding SNAPPY compression.

The access pattern for rows and columns is completely random. Any 500 rows and 30 cells can be requested at any time. There is no easy way to group the rows and columms for faster access.

I'm still seeing a latency of up to 5 seconds, which is much too slow. Is this too fast a response time to expect from such a large dataset? Or am I making some basic error? Should I try encoding and compressing the row key?