I have been engineering and developing a NoSQL database engine (using C#.Net) for .Net technologies. Similar to the available NoSQL database, I store all documents in JSON format, and I keep all related document records (i.e. Album1, Album2, Album3) in a single file.
I have been testing the solution to see its actual performance in a real world scenario, and using Visual Studio Unit Testing framework, my solution has managed to query and search 2,000,000 documents in approximately 12 seconds.
My first question is, how good is that performance if it is a good result at all?
Secondly, since all records are eventually saved in files, I have implemented a singleton design pattern that buffers up all physical documents in memory in order to prevent the need for concurrent file processing. At 2,000,000 and assuming there are 10 document categories (Album, Genre, Artists,...) this will consume more than 8GB of RAM which is not good. On the other hand, if I disable the buffering and make all queries dependent on searching the physical files, assuming that I get 100 concurrent request and each take 12 seconds to complete, these 100 request may take up to 1,200 seconds to complete which is I think is terrible.
The second question is, how could this be optimized? I mean, a NoSQL database, unlike a SQL database, is meant to preserve massive data, and such massive data cannot be fully buffered in RAM or be searched over and over on disk. In theory, how should this be implemented?