I am new to the community, so feel free to correct any clumsy mistakes.

I have a large set of pdf and docx files ~10TB. I want to perform searches for specific words that will yield the document, the page and the line containing such word.

As a naive approach I wrote some simplistic code on python:

    import PyPDF2
    import os
    
    def line_matches(line, s_terms):
        return any(ele in line for ele in s_terms)
    
    directory = ""
    files = os.listdir(directory)
    search_term = 'idea'
    
    for file in files:
        if (file[-4:] == '.pdf'):
            output = open(file_name[:-4] + '.txt','w')
            pdfReader = PyPDF2.PdfReader(directory + '/' + file)
            for i, p in enumerate(pdfReader.pages):
                lines = p.extract_text().splitlines()
                for j, l in enumerate(lines):
                    if line_matches(l, search_terms):
                        output.write("page " + str(i + 1) + ", line " + str(j + 1) + ": " +
                                 l.encode('utf-8', 'ignore').decode('utf-8') + '\n')

This code works, however it is awfully slow. What is a good approach to make the process faster?
I tried multithreading, which obviously performs better, but I still feel there's a better way of doing it.

 - Should I change programming language and use a better library like [XpdfReader][1]?
 - Should I preprocess the files, should I create data structures to better store the documents?
 - Is building a database for these files worth it in terms of speed? what if I get more files (1PB)?

My end goal is to have a user interface that will allow any user to find in the fastest way the location in a document where his query occurs. Thank you in advance for the replies


  [1]: https://www.xpdfreader.com/