I have an application requirement to store a large amount of dictionaries in a database for fast reading. I know beforehand the approximate size and number of data that needs to be stored. The requirement is as follows:

  • Store around 100.000 dictionaries or hash tables (for fast access to each value in dictionary)
  • Each dictionary/hash table can have up to 10.000 key/value pairs

Why am I thinking of dictionaries? Because I need lightning fast reads, I can cope with slow writes, as the writes are going to happen offline, but data needs to be read as fast as possible (used for a search engine). My first thought was to use a key/value store, because each dictionary has a unique keyword associated to it. The following is an example of the data that needs to be stored and read efficiently:

"house": {
             "12345322": 0.467,
             "33627312": 0.657,
             "88349823": 0.342,
             ... (up to 10.000 unique keys here)

"car": {
           "12345322": 0.378,
           "33627312": 0.851,
           "88349823": 0.266,
           ... (up to 10.000 unique keys here)
... up to 100.000 dictionaries (unique keys)

Currently I am looking for the right tool to solve this problem. What database system do you recommend? I have a pair of tools that come to mind (Redis) but I do not know if they are adequate for this amount of data.

  • Wjhy not a RDBMS? – ypercubeᵀᴹ Oct 16 at 18:16
  • I thought about it but to store this data in a SQL table it would have 100.000 * 10.000 = (1 billion) rows. Don't know how this would perform even with proper indices – flizana Oct 16 at 19:41
  • 1
    If you store it in Redis, it would still be 1 billion key/values pairs, wouldn't it? – ypercubeᵀᴹ Oct 16 at 19:47
  • @ypercubeᵀᴹ yes, it would still be 1 billion key/value pairs – flizana Oct 16 at 20:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.