Best approach to search for a word in a corpus of pdf and docx files

Question

I am new to the community, so feel free to correct any clumsy mistakes.

I have a large set of pdf and docx files ~10TB. I want to perform searches for specific words that will yield the document, the page and the line containing such word.

As a naive approach I wrote some simplistic code on python:

import PyPDF2
import os

def line_matches(line, s_terms):
    return any(ele in line for ele in s_terms)

directory = ""
files = os.listdir(directory)
search_term = 'idea'

for file in files:
    if (file[-4:] == '.pdf'):
        output = open(file_name[:-4] + '.txt','w')
        pdfReader = PyPDF2.PdfReader(directory + '/' + file)
        for i, p in enumerate(pdfReader.pages):
            lines = p.extract_text().splitlines()
            for j, l in enumerate(lines):
                if line_matches(l, search_terms):
                    output.write("page " + str(i + 1) + ", line " + str(j + 1) + ": " +
                             l.encode('utf-8', 'ignore').decode('utf-8') + '\n')

This code works, however it is awfully slow. What is a good approach to make the process faster? I tried multithreading, which obviously performs better, but I still feel there's a better way of doing it.

• Should I change programming language and use a better library like XpdfReader?
• Should I preprocess the files, should I create data structures to better store the documents?
• Is building a database for these files worth it in terms of speed? what if I get more files (1PB)?

My end goal is to have a user interface that will allow any user to find in the fastest way the location in a document where his query occurs. Thank you in advance for the replies

Answer 1

Brute force search of terabytes is folly.

Use a database with a "FULLTEXT" index. Convert the PDFs to text to store the text without all the PDF baggage. (Keep a link to the pdf file if you like.)

MySQL's (for example) FULLTEXT can query for an uncommon word (or group of words) quite rapidly.