When dealing with very large collections i came across timeout issues when querying and inserting. Even though the collections were indexed and optimized to combine small documents together.
The rate of insertion for a given collection is 172,800 documents per day.
I solved this by partitioning the DB into months.
ASDF -> ASDF_10_2018 , ASDF_11_2018 ... ASDF_05_2020
by doing so the collection size under each db was decreased and i did not encounter any query or insertion issues.
The things is that there are now significantly more collections. Orignally 3000 collections are now 3000 collections each month.
MongoDb now as an issues with the number of open files it has to hold in order to maintain my partitioned DB's and collections.
1) Is there a better way then the portioning described above of solving the problems that occurred due to the large collections.
2) Is there a way to tell mongo to not open so many files. Most of the Db's are old and not commonly queried and can stay dormant on my part, is there a way to tell Mongo not to open the files related to them ?