0

I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each tLD such as .com, .org etc.

During the initial insertion of the records( such as the quick insertions of the lest populated and obscure tLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.

However this slows to a crawl, part way through. Since MongoDB uses a B-tree for indexing primaries, shouldnt the performance slowdown due to database size be minimal ?

Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself

|improve this question
  • Is replication slowing it down? What is your write concern set to? – SqlWorldWide Aug 16 '17 at 1:05
  • How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown? – Stennie Aug 16 '17 at 2:50
  • My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million) – Abhinav Vishak Aug 16 '17 at 15:24
0

You have run out of resources!

With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.

Can you (for start) increase memory to (at least) 16 - 32GB?

|improve this answer
  • I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ? – Abhinav Vishak Aug 17 '17 at 20:41
  • also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue. – Abhinav Vishak Aug 17 '17 at 20:42
  • There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits. – JJussi Aug 18 '17 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.