Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

There are 100 data files, each with 60 fields and over 4 million records. There is a perl program that inserts the records or updates them based on an userdefined _id field. There is also a History collection that stores all values ever written for three fields. A replica set with two servers and an arbiter has been set up. Initially the files were loading into the MongoDB at 45 minutes per file. After around 20 files the speed has dropped considerably. The speed at this time is 20 hours per file. The servers have started slowing down badly. I am unable to use the logout command even quickly.

I have built indexes on the _id field with hashed indexing and for the History collection I have built indexes with id and date field. The number of records at this time in the collections are 4 million for the actual data collection and around 100 million for the History collection. I have two 17 GB RAM processor, of which only around 3.5 gigs are used as per the Mongostat res command. However since the data records are to be inserted date wise sequentially, I cannot exploit parallelism either.

The limits of mongo for the specific scenario have been reached? Is this slowdown to be expected? I have fsynced manually every now and then to ensure files are being written to disk. Is there some other diagnostics that I can run to better explain the situation? Is there a solution to this?


share|improve this question
Can you run mongostat when inserts are in progress? If you can - run it for 5-10 minutes, put output somewhere like pastebin and add a link here. –  diversario Aug 18 '13 at 1:03
Thanks, I currently cannot run that because I am no longer doing the insertion. From what I understood, it turns out that MongoDB is very fast when the working set fits in RAM. However when it doesn't it starts showing signs of slowing. A better performance was observed when RAM was increased, but not significant enough to warrant any encouragement :). –  Sai Aug 28 '13 at 4:47
You might want to use bulk operations during insert. In case not all your RAM is used, it is safe to assume that either the disks or the inserting program is the limiting factor. With bulk ops, you speed up things for both limiting factors. There is something seriously wrong. Unless your fields are very big, 200k inserts/h are a mere joke. You want to have your code peer reviewed and make sure that IO operations are efficient. See the production notes for further details on this. –  Markus W Mahlberg Dec 19 '14 at 7:32
@MarkusWMahlberg thanks for the response. Bulk inserts would not be possible in this case because of the fact that I would have to look at each record and classify it as 'insert' or 'update' and only then I can do the operation. Also the insertion would need to be in order because I would like to have the history of updates. There is no bulk upsert as far as I know in MongoDB. Also with respect to the Perl Program being inefficient, it is possible, however I doubt, because of the terrific slow down from few seconds to few hours. It smells like a performance limit of Mongo. What would u suggest –  Sai Dec 19 '14 at 16:04
There are bulk upserts: bulk.find({...}).update({...},{upsert:true}). Furthermore, you can do var bulk=db.collection.initializeOrderedBulkOp(). –  Markus W Mahlberg Dec 19 '14 at 16:44

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.