I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each tLD such as .com, .org etc.
During the initial insertion of the records( such as the quick insertions of the lest populated and obscure tLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.
However this slows to a crawl, part way through. Since MongoDB uses a B-tree for indexing primaries, shouldnt the performance slowdown due to database size be minimal ?
Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself