0

I've asked the question on here: https://stackoverflow.com/questions/71819288/mongodb-vs-elasticsearch-indexing-parallel-arrays

But, I didn't get any answer. I believe here it's the right place for this. Currently I'm using MongoDB and my documents includes 2+ array fields and my query needs to filter based on both fields. In MongoDB it's not possible to index more than 1 array field, therefore I may need to change the data model (although that may have some other drawbacks since joining is too costly) or use something like ElasticSearch nested models, and do exact filtering?

Here it is an example.

 { 
    "_id" : ObjectId("55b73bb3f1459e2ae3e7dadb"), 
    "segments" : [
        NumberInt(335), 
        NumberInt(533), 
        NumberInt(16143), 
        NumberInt(16206), 
        NumberInt(16207), 
        NumberInt(22012), 
        NumberInt(22014), 
        NumberInt(22050), 
        NumberInt(22096), 
        NumberInt(22272), 
        NumberInt(22537), 
        NumberInt(22563), 
        NumberInt(22564), 
        NumberInt(22571)
        ...rest (more than 10k records)
    ], 
    "locals" : [
        ObjectId("someid"), 
        ObjectId("someid"), 
        ObjectId("someid"), 
        ObjectId("someid"), 
        ObjectId("someid"), 
        
        ...rest (more than 10k records)
    ], 
    validFrom,
    validUntil,
    ...rest
}

From the above model, the aim is to find the documents that matches the following:

Let assume we have 100+ available locals object ids for us

Let assume we have 50+ segments available for us

Find all records that matches the conditions and it’s between valid dates and assume that there are 50k+ existing documents like this.

8
  • 1
    One reason you're not gathering much response (on either site) is you've provided very little information on your data. It's pretty much impossible to advise anything with the given information so far. Please update your post to give some examples of your data, clarification on what it represents, how you currently have it structured / how it relates to itself, and some concrete information on your use cases with it. Elasticsearch is a search engine, not a traditional DB system (as mentioned in the answer you linked in your other post). I suspect a SQL RDBMS may even be applicable here.
    – J.D.
    Apr 20 at 11:58
  • OK, I've just updated the question. Thanks. I know that Elasticsearch is search engine but it is also can be used for filtering as in here: blog.quarkslab.com/… Since there are too many records, and join operations are costly, that's why I've asked also to understand more about how do they differ. cs.stackexchange.com/questions/150568/…
    – BySpecops.
    Apr 20 at 12:24
  • Thanks! Yes they're all tools that can interchangeably be used for many of the same jobs, but that doesn't mean they're the best tool. Excel can filter data too, but it's a terrible database tool. Elasticsearch is best at full text type of searching / filtering. "too many records, and join operations are costly" - This is subjective. Join operations in a traditional SQL RDBMS take anywhere from 10s to 100s of milliseconds. Most people can tolerate their read queries taking an extra 100 milliseconds. I've joined Tables with billions of records in under a second. Usually it's a non-issue.
    – J.D.
    Apr 20 at 12:59
  • You're right! That's why I was trying to understand how ElasticSearch indexing differs from traditional B-Tree indexes. Or is it differ at all. I'm not a ElasticSearch fan I was just read some articles and benchmarks and thought that it might be a good candidate for such an application. Since as I mentioned before, in MongoDB it's not possible to index this kind of collection due to there are multiple array field. I'm OK to discuss different data models. I'm just curious about what would be the best data structure and tool for such a query.
    – BySpecops.
    Apr 20 at 13:08
  • Btw, I've tried several times to use MongoDB's $lookup feature I cannot say that my experience is the same as yours on RDBM's. Therefore, I couldn't think that as an option for this example.
    – BySpecops.
    Apr 20 at 13:13

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.