A question about retrieving performance for simple but bug key-value DBs on python3.

I am working on a pipeline to compare several entries based on key-values (genetic variants found in patients with known variants in different DBs of several GBs for each chromosome) on python3. The most important thing would be performance in retrieving information (as I need to write the DB only once and then just querying it, no updating).

I tried SqliteDict but it performed not so well (sqlitedict retrieve operations performance), I also tried MongoDB but it does not seemed to help. I had a look at redis but it can not be the case due to the DBs dimension and redis in-memory type.

Other ideas to store very big DBs (several GBs) in a key-value NoSql (supposedly faster than Sql) dictionary that can be later vastly retrieved (I hoped SqliteDict to be faster).

Thanks a lot for any help!

  • Qdbm and Family made for mostly KV lookups, maybe? It has a python binding. You did however what is big in your case. The overall number of entries, their key or their values? – eckes Mar 18 at 20:58
  • Thanks a lot, the entries would be 1-1.5*10E+6 per chromosome (and there are 24 chromosome) and this would be just a branch with others even bigger. The key can be a integer of the position on the chromosome (ex: 2088081) or a code with chromosome number before like (1-2088081). Values would be a list of 1000 characteristics for each position, all short alphanumeric strings. I am giving a try with MongoDB in this moment but it seems to generate huge files. – cccnrc Mar 18 at 21:59
  • When it comes to performance, actual examples are necessary. I.e., show the actual query you want to optimize, with actual values. Note: to edit the question, click "edit". – CL. Mar 19 at 8:58
  • So 10 Byte Keys, 1 MB value and you only look up single keys or ranges as well? You don’t have to search on values? Would be good if you can add those details to your question. Also aren’t there systems for exactly that usecase already in bio informatics frameworks? – eckes Mar 20 at 10:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.