We have a fairly beefy server (15 GB RAM, 2 cores) running a reasonably large MongoDB instance. The Mongo instance hosts two separate databases that communicate via application code. The larger database (let's call it DB1) contains on the order of 100 million records with more being added all the time.
The second DB (let's call it DB2) contains about 10 million records and grows much more slowly (there are a lot of writes, but most are updates).
Recently, we noticed that reads on DB2 have become ridiculously slow, taking up to 2-3 minutes to simply loop through about a 1000 records. On profiling the query, it looks like the right indexes are set and are being used, but that the performance for some as-yet unknown reason has completely fallen off a cliff.
Considering that DB2 doesn't grow much, could this be because DB1 is affecting the performance (maybe by competing for RAM)? Is there a more direct mechanism for two databases to interfere with each other?
I'm trying to identify the bottleneck here, so any pointers in the right direction would be much appreciated.
I should also note that while DB1's performance has also degraded a bit, it hasn't completely fallen off a cliff like DB2.