I have a set of n sentences (n >= 100k).

I want to perform queries of the sort where given a set of n_i sentences i return a set of m_i words that is present in one subset of n_i sentences and not present in the other.

Which database is appropriate for my use case. I have looked into sqlite using sqlalchemy and i havent been able to quite figure out how to store these m words to each n strings.

Also i prefer something that has python apis . Should i be using graph databases. I am a total noob at this stuff so any help would be greatly appreciated

closed as off-topic by Josh Darnell, Philᵀᴹ, Vérace, John Eisbrener, kevinsky Apr 16 at 12:07

This question appears to be off-topic. The users who voted to close gave this specific reason:

  • "Shopping list question - questions about which tool, library, product or resource you should use are off-topic here because they quickly become obsolete and often are just about the preferences of the answerer. If you have an issue with or a question about a specific tool, please revise your question to conform to that scope." – John Eisbrener, kevinsky
If this question can be reworded to fit the rules in the help center, please edit the question.

  • 10^10 words? seriously? – Akina Apr 14 at 19:07
  • @Akina - I think what the OP means is that the 10 ^ 10 are the (potential?) substrings of length < 50 within the 100k strings? – Vérace Apr 14 at 20:36
  • Perhaps if you gave us some sample data, (maybe a bit smaller than 100k), of your input, your expected output and the logic you used to get that output, we might be able to help. In the Open Source sphere, I would recommend that you look at PostgreSQL all the time! – Vérace Apr 15 at 11:34
  • This question will likely be closed as it's the quintessential shopping list question, meaning you're looking for a product or list of potential products that will solve your problem. We often close these as these lists often get outdated quickly and this question quickly loses relevance as a result. If you have a more specific issue in mind though, please edit your question accordingly. – John Eisbrener Apr 15 at 13:08

This sounds more like a programming problem than a data storage and retrieval problem. It probably won't matter where you store the data. You could just use something like Azure Blob storage for the data, and provision a Spark cluster to process the data at scale, using Python. - David Browne

If you want an Open Source database, then go for PostgreSQL - you will find that it's infinitely more capable than MySQL for... well, virtually everything! You might be able to do some sort of CROSS JOIN (LATERAL) between your 50-mers using LAG and proceed from there.

Using joins is pure speculation on my part. Python would probably be better for this With respect to David Browne's suggestion, a programming solution may be required. Having said that, SQL (+ WITH RECURSIVE clause) is Turing complete, so in theory, anything's possible.

I suggest that you start a new question with some sample data. Put that sample data in dbfiddle.co.uk (or similar). The sample data doesn't have to be the same size as your real data. To start with, think of it as a proof of concept, so that people can understand the core of the problem - for this, actual size is unimportant. Use, say, sequences of 100 and 5-mers as an example? Concentrate on asking about the SQL problem rather than which server to use - you could though state a preference for Open Source? Python works with everything! - Vérace

Not the answer you're looking for? Browse other questions tagged or ask your own question.