I am running a simple iterator over all the documents in a collection in a mongo replica set (primary, secondary, arbitrary). On a collection with 1.000.000 items it takes about 10 seconds. On a collection with 12.000.000 it takes more than one hour. We have seen cases when it takes 6 hours to iterate over such a volume.
Why is this? I read somewhere that more than 2.000.000 items in a collection is bad for performance. Is that correct?
This is on MongoDB 3.0 on a RedHat VMware machine.
The size of the collection is some 100GB while the database itself is 400GB. There are some 40 smaller collections in the same database. The average size of a document in this collection is 8KB. Memory usage seems to be around 6GB of 8GB available on the machine. I haven't tracked memory usage while actually running the query.
There may be some updates on the collection while it is running.