0

I'm trying to create a search algorithm that will return more diverse results based on one common field in my "Articles" collection in my MongoDB.

My "Articles" schema basically looks like this:

const authorSchema = {
  title: string,
  content: string,
  author: ObjectID, // the id of an author in the Author's collection
  publisher: ObjectID // the id of a publisher in the Publisher's collection
};

"Articles" might have 10^6 objects and "Publishers" might have something like 20 with each individual publisher being responsible for a differing amount of articles. Currently if I query the Articles collection with some arbitrary filter, limiting the results to 50 objects, that doesn't mention the publisher field I might get a list of results from one publisher. I want to diversify my search results based on how represented each publisher is in my "Articles" collection.

This is the search algorithm I've come up with so far. Assume all fields are valid:

// Publisher is a mongoose model representing all publishers

const MAX_RESULTS = 50;

var PublisherWeights: Array<{ _id: ObjectId; count: number }>;

PublisherWeights = // This algorithm is meant to calculate how many articles should be returned for each publisher based on the proportion of articles who's `publisher` field is equal to that publisher. It's a weighted average algorithm.
  PublisherWeights ||
  (PublisherWeights = await (async () => {
    const publishers = await Publisher.find({});
    const articleSum = publishers.reduce(
      (sum, pub) => sum + pub.stats.articles, // `pub.stats.articles` is the sum of all the articles linked to this publisher in the "Articles" collection
      0,
    );

    const weights = publishers.map((pub) => ({
      _id: pub._id,
      count: Math.ceil((pub.stats.articles / articleSum) * MAX_RESULTS),
    }));

    return weights;
  })());

const results = await Promise.all( // This takes waaaaayyy to long
  PublisherWeights.map(async (w) =>
      Article.find({title: new RegExp(term)) // `term` is a string inputted by the query api. Not necessary for this example
        .find({ publisher: w._id } as any) // `as any` is necessary to suppress type warnings 
        .limit(w.count), // `w.count` tends to be between 1 to 10
  ),
);

const flatResults = results.reduce((flat, curr) => flat.concat(curr), []);

What I find is that this algorithm is incredibly slow (compounded with the fact that I'm cheaping out on my database host). It can take up to 3 seconds to return results. I suspect that it's because I'm hitting a massive collection with narrow limits, and the potential for 20+ queries running in parallel. I want to be able to run these queries in a bulk operation but I can't find "bulk querying" in the nodejs mongodb driver documentations or the mongodb API. I'm also concerned about using Regex to find partial matches.

I have added an index in my "Articles" collection for the publisher field but I'm not sure if it's helping.

I do plan on implementing redis layer database caching, but that doesn't solve the edge case for new queries long term. And before I consider throwing more money at my database host for faster queries I'd like to optimize what I have as best as I can.

I'm new to database design. I've never created a search algorithm like this. Can anyone recommend improvements I can make?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.