I am running a simple iterator over all the documents in a collection in a mongo replica set (primary, secondary, arbitrary). On a collection with 1.000.000 items it takes about 10 seconds. On a collection with 12.000.000 it takes more than one hour. We have seen cases when it takes 6 hours to iterate over such a volume.

Why is this? I read somewhere that more than 2.000.000 items in a collection is bad for performance. Is that correct?

This is on MongoDB 3.0 on a RedHat VMware machine.

The size of the collection is some 100GB while the database itself is 400GB. There are some 40 smaller collections in the same database. The average size of a document in this collection is 8KB. Memory usage seems to be around 6GB of 8GB available on the machine. I haven't tracked memory usage while actually running the query.

There may be some updates on the collection while it is running.

  • Hi, could you add some details on the collection size and the vm memory? Also, is anyone else might be adding or changing the content while you read? – cohenjo Oct 17 '16 at 14:08
  • Updated the question! – froderik Oct 18 '16 at 6:03
  • from the sizes above it looks like the server will have a lot of work reading pages from disk and clearing the mapped areas to free up space (the small query with 1M docs probably mostly fits the RAM so less swapping of pages). If you could attache the cursor explain it will help identify the bottle neck. – cohenjo Oct 18 '16 at 8:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.