For non-incremental, but overlapping, imports, I worry about upserts to be very slow and locking the resources for hours.

From different sources, I receive 1 json dump per day per source for up to 3 days. Sometimes, they arrive 1-2 days delayed. That's why every day, all of them - if available - are re-imported into the "merged" collection. This is done by upsert to make sure that e.g. yesterdays documents, that have been imported and also processed and sometimes updated, won't be overwritten.

The input data from different sources is sort by date, but there is not a single unique field.

The merged collection where all the data becomes imported into, 5 indexes, which seems to make the upsert/import even slower.

Each document has a (non-unique) unix timestamp value that has an index, too (not the mongodb date/timestamp, but a number). It feels like there was no advantage by the ordered data and the upsert looks up the entire index although a unix timestamp exists in each document.

Is there a better practise or at least an option to increase the speed of imports of this non-incremental but sorted data by taking advantage of the unix_timestamp field/index instead of seeking for former documents in the entire collection?

  • Can you comment on how you are structuring the import? Are you using the mongoimport tool (with --upsert/--upsertFilds) or have you written your own script to do the import?
    – Adam C
    Jun 19, 2013 at 11:19
  • mongoimport -d ... -c ... --upsert < .../path.json
    – ledy
    Jun 19, 2013 at 12:58

2 Answers 2


Since you are using mongoimport to do these upserts without --upsertFields, it will be using the _id index to do the upsert (see the note in the docs). That means that it will be scanning that index, and if you are including the _id field in the json dump that should be fine in terms of that particular search as long as that index is in memory. If you are not including the _id, then that will mean a full scan.

You can alter this behavior by using the aforementioned upsertFields option, though you will want to make sure those fields you pick are indexed (a compound index of the fields used rather than _id).

The overhead from the other indexes is actually probably just the overhead that occurs when you are updating several indexes as part of a data load.

Finally, if you are doing this repeatedly and intend to keep going, I would recommend creating your own tool to do this and having more control over the entire process. While mongoimport is a fine choice for simple tasks, more complex imports are better off handled by a more complex and customizable tool.

  • so mongoimport is using the fastest import method already, scanning _id index "only"? Anyways, it's locking the db as soon as i start mongoimport: nopaste.info/d4d8009b5a.html The RAM is 50% only and the index is checked/OK. Can't find the mistake or reason for locking.
    – ledy
    Jun 19, 2013 at 15:02
  • Assuming you are running mongostat with the defaults, then that is doing ~75 updates a second, and there are page faults, meaning that something it is looking for is not in fact in RAM - don't forget that if it finds the document, it will have to load it into memory in order to determine how it must be updated. If you have disk writes and reads competing, along with a lot of updates.....well it is going to be slow.
    – Adam C
    Jun 19, 2013 at 15:21

When you have a CSV file and that has 6GB of data over crores of record still your --MONGOIMPORT will work very fine instead what you have to do is split that file using any powershell ps1 then import it inorder to make your process faster , but there is no choice of having the slow performance with mongoimport.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.