I have a "hot" MongoDB(4.0.5) capped collection with a timestamp (and several other) indexes. Many applications (Java, NodeJS & C++) insert (never update) records into this collection at a rate of ~50 inserts/second. Normally these inserts are very quick and take only a few milliseconds.
When I run a map-reduce query on the most recent 5 minutes of data, calls to insert new records into the source collection are blocked or delayed. New inserts during the map-reduce call can be delayed > 10 seconds which is roughly the time that it takes to run the map-reduce operation. The output from the map-reduce operation is written to the same database.
What can I do to tune MongoDB or the map-reduce operation so that inserts can continue to flow into the capped collection without delay while the map-reduce operation runs? I need to get the summary analysis in near real-time so I don't want to wait until the insert load is reduced before running the analysis.
I do not believe that this is a system level resource constraint. I have monitored CPU, network, i/o and memory during the map-reduce call and CPU idle time remains > 50% and network and i/o latency remains low. IO/wait times remain under 5%.
