5

There are 100 data files, each with 60 fields and over 4 million records. There is a perl program that inserts the records or updates them based on an userdefined _id field. There is also a History collection that stores all values ever written for three fields. A replica set with two servers and an arbiter has been set up. Initially the files were loading into the MongoDB at 45 minutes per file. After around 20 files the speed has dropped considerably. The speed at this time is 20 hours per file. The servers have started slowing down badly. I am unable to use the logout command even quickly.

I have built indexes on the _id field with hashed indexing and for the History collection I have built indexes with id and date field. The number of records at this time in the collections are 4 million for the actual data collection and around 100 million for the History collection. I have two 17 GB RAM processor, of which only around 3.5 gigs are used as per the Mongostat res command. However since the data records are to be inserted date wise sequentially, I cannot exploit parallelism either.

The limits of mongo for the specific scenario have been reached? Is this slowdown to be expected? I have fsynced manually every now and then to ensure files are being written to disk. Is there some other diagnostics that I can run to better explain the situation? Is there a solution to this?

Thanks

Improve this question
6
  • Can you run mongostat when inserts are in progress? If you can - run it for 5-10 minutes, put output somewhere like pastebin and add a link here.
    – diversario
    Aug 18, 2013 at 1:03
  • Thanks, I currently cannot run that because I am no longer doing the insertion. From what I understood, it turns out that MongoDB is very fast when the working set fits in RAM. However when it doesn't it starts showing signs of slowing. A better performance was observed when RAM was increased, but not significant enough to warrant any encouragement :).
    – Sai
    Aug 28, 2013 at 4:47
  • You might want to use bulk operations during insert. In case not all your RAM is used, it is safe to assume that either the disks or the inserting program is the limiting factor. With bulk ops, you speed up things for both limiting factors. There is something seriously wrong. Unless your fields are very big, 200k inserts/h are a mere joke. You want to have your code peer reviewed and make sure that IO operations are efficient. See the production notes for further details on this.
    – Markus W Mahlberg
    Dec 19, 2014 at 7:32
  • @MarkusWMahlberg thanks for the response. Bulk inserts would not be possible in this case because of the fact that I would have to look at each record and classify it as 'insert' or 'update' and only then I can do the operation. Also the insertion would need to be in order because I would like to have the history of updates. There is no bulk upsert as far as I know in MongoDB. Also with respect to the Perl Program being inefficient, it is possible, however I doubt, because of the terrific slow down from few seconds to few hours. It smells like a performance limit of Mongo. What would u suggest
    – Sai
    Dec 19, 2014 at 16:04
  • 1
    @MarkusWMahlberg ok that sounds good. If you can post this and the other two comment as answer, I would be glad to accept. I think the fact that searching will occur every step (whether it's bulk or not) is the reason for slowness because everytime I would have to locate that record in a set of 1 million and then insert it and then again repeat the search for every insert. What do you think?
    – Sai
    Dec 19, 2014 at 16:48

1 Answer 1

Reset to default
3

Community wiki answer generated from comments on the question by Markus W Mahlberg:

You might want to use bulk operations during insert. In case not all your RAM is used, it is safe to assume that either the disks or the inserting program is the limiting factor. With bulk ops, you speed up things for both limiting factors. There is something seriously wrong. Unless your fields are very big, 200k inserts/h are a mere joke. You want to have your code peer reviewed and make sure that IO operations are efficient. See the production notes for further details on this.

There are bulk upserts: bulk.find({...}).update({...},{upsert:true}). Furthermore, you can do var bulk=db.collection.initializeOrderedBulkOp().

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.