1

I'm trying to insert Whois Records ( domain name and relevant info) into MongoDB after parsing them from CSV files. I currently have a collection for each TLD such as .com, .org, etc.

During the initial insertion of the records( such as the quick insertions of the least populated and obscure TLDs and initial .com etc insertion ), I get decent insert speeds of about 10k-15k inserts per second.

However, this slows to a crawl, partway through. Since MongoDB uses a B-tree for indexing primaries, shouldn't the performance slowdown due to database size be minimal?

Extra Information if relevant: MongoDB 3.4, WiredTiger, Java 1.8, primary index is the domain name itself

3
  • Is replication slowing it down? What is your write concern set to? Commented Aug 16, 2017 at 1:05
  • How much RAM do you have? How large is the data set when your insertions start to slow down? A likely suspect for slowdown over time would be your data set becoming constrained by RAM or I/O. Do you have any other metrics to correlate with the period of slowdown?
    – Stennie
    Commented Aug 16, 2017 at 2:50
  • My VM is set to 8 GB currently, I don't have anything running on the server except MongoDB( though admittedly the i have to insert about 202 million records across 1000 collections, and the .com collection itself is about 133 million) Commented Aug 16, 2017 at 15:24

1 Answer 1

0

You have run out of resources!

With 202M records, even one record would be only 64 bytes (you can check with db.collection.stats().avgObjSize), the size is 12GB and then comes index. The optimal situation is when active data, (index and data) is in the memory. This case it is NOT. So, mongod starts using disk heavily and if your disk is not high IOPS (SSD base) and low latency base, your machine starts slowing down.

Can you (for start) increase memory to (at least) 16 - 32GB?

3
  • I think i may have some additional info that is relevant. I reran the insert operations, and while monitoring with mongostat I noticed that the performance dropped when the RES/VIRT memory usage was nearing the wiredtiger default limit of 50%-1GB( around 8 GB, i had increased my ram to 16gb ). how would i go about increasing the mongod limits ? Commented Aug 17, 2017 at 20:41
  • also restarting mongod midway seems to "flush" the ram causing no problems, so it def looks like a RAM issue. Commented Aug 17, 2017 at 20:42
  • There are no real mongo limits here. Mongo will always use (eventually) all memory what is available (period). More memory is better. Until you reach the limit where all data and indexes fit in the memory and after that increasing memory don't bring any benefits.
    – JJussi
    Commented Aug 18, 2017 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.