-4

I have a set of n sentences (n >= 100k).

I want to perform queries of the sort where given a set of n_i sentences i return a set of m_i words that is present in one subset of n_i sentences and not present in the other.

Which database is appropriate for my use case. I have looked into sqlite using sqlalchemy and i havent been able to quite figure out how to store these m words to each n strings.

Also i prefer something that has python apis . Should i be using graph databases. I am a total noob at this stuff so any help would be greatly appreciated

Improve this question
4
  • 10^10 words? seriously?
    – Akina
    Apr 14, 2019 at 19:07
  • @Akina - I think what the OP means is that the 10 ^ 10 are the (potential?) substrings of length < 50 within the 100k strings?
    – Vérace
    Apr 14, 2019 at 20:36
  • Perhaps if you gave us some sample data, (maybe a bit smaller than 100k), of your input, your expected output and the logic you used to get that output, we might be able to help. In the Open Source sphere, I would recommend that you look at PostgreSQL all the time!
    – Vérace
    Apr 15, 2019 at 11:34
  • This question will likely be closed as it's the quintessential shopping list question, meaning you're looking for a product or list of potential products that will solve your problem. We often close these as these lists often get outdated quickly and this question quickly loses relevance as a result. If you have a more specific issue in mind though, please edit your question accordingly.
    – John Eisbrener
    Apr 15, 2019 at 13:08

1 Answer 1

Reset to default
0

This sounds more like a programming problem than a data storage and retrieval problem. It probably won't matter where you store the data. You could just use something like Azure Blob storage for the data, and provision a Spark cluster to process the data at scale, using Python. - David Browne

If you want an Open Source database, then go for PostgreSQL - you will find that it's infinitely more capable than MySQL for... well, virtually everything! You might be able to do some sort of CROSS JOIN (LATERAL) between your 50-mers using LAG and proceed from there.

Using joins is pure speculation on my part. Python would probably be better for this With respect to David Browne's suggestion, a programming solution may be required. Having said that, SQL (+ WITH RECURSIVE clause) is Turing complete, so in theory, anything's possible.

I suggest that you start a new question with some sample data. Put that sample data in dbfiddle.co.uk (or similar). The sample data doesn't have to be the same size as your real data. To start with, think of it as a proof of concept, so that people can understand the core of the problem - for this, actual size is unimportant. Use, say, sequences of 100 and 5-mers as an example? Concentrate on asking about the SQL problem rather than which server to use - you could though state a preference for Open Source? Python works with everything! - Vérace

Improve this answer
0

Not the answer you're looking for? Browse other questions tagged or ask your own question.