For non-incremental, but overlapping, imports, I worry about upserts to be very slow and locking the resources for hours.
From different sources, I receive 1 json dump per day per source for up to 3 days. Sometimes, they arrive 1-2 days delayed. That's why every day, all of them - if available - are re-imported into the "merged" collection. This is done by upsert to make sure that e.g. yesterdays documents, that have been imported and also processed and sometimes updated, won't be overwritten.
The input data from different sources is sort by date, but there is not a single unique field.
The merged collection where all the data becomes imported into, 5 indexes, which seems to make the upsert/import even slower.
Each document has a (non-unique) unix timestamp value that has an index, too (not the mongodb date/timestamp, but a number). It feels like there was no advantage by the ordered data and the upsert looks up the entire index although a unix timestamp exists in each document.
Is there a better practise or at least an option to increase the speed of imports of this non-incremental but sorted data by taking advantage of the unix_timestamp field/index instead of seeking for former documents in the entire collection?