I have a use case where I have access to a large database with many documents (millions+). I want to build functionality such that, when I want to add a new document to that database, I first want to scan the database for potential duplicate documents. Documents may differ in file type and other information (i.e. one is a digital copy, while another is a hard copy that was scanned in via a printer) while still materially being the same document, so I unfortunately can't rely on metadata or similar.
In order to determine if a document is a duplicate, I plan to use some weighted combination of NLP (via something like spaCy) to compare document similarity, comparison of word distributions (both the simple word distribution and something like TF-IDF), and other relevant metrics.
In order to actually find potential duplicates for a newly uploaded document, I can't think of any way to avoid scanning every document in the database, comparing one-by-one, and tracking the one with metrics that matched most closely.
Thoughts I've had for optimizing this:
- I know indexing can often be used to speed up search operations, but to my understanding that's only good for searching for a specific value in a specific column. I don't think it's a good fit, as I'm trying to essentially take a weighted average of each metric and report which are closest. I guess I could index every column; could this potentially be worth it, or would the constant reindexing be a huge performance penalty?
- I've been thinking of using a clustering (unsupervised machine learning) model in order to cluster similar documents together, then try and use the clusters it gives me to try and determine which cluster the new document would fit in, then just search in there, but I'm getting caught up in the details. I'm sure this would be a practical approach for finding preexisting duplicates in the database, but is this practical to do every time a new document is added to the set (i.e. could it be used to speed up the actual search through the database)? I'm not too well-versed in machine learning, so I'd appreciate some input on this.
So ultimately - is there a way for me to structure my database such that I don't need a linear search in this scenario?