1
  • I have been massively frustrated by the erratic behaviour of Mongodb's bulk inserts. I'm trying to write 20K documents in a bulk-insert query. Sometimes the inserts take as low as 5 seconds and sometimes (for no change in environment) takes 300 to 400 seconds. This is massively frustrating as it leaves the application architect in dark as to have any assumption on time requirement of this query.

  • I have nearly 6 million documents (tweets) in a collection and I have several indices (including text index) on collection. The overall size of indices is around 7Gb on a 14 Gb machine. Journaling is off.

  • I have realized that writes (inserts) are super slow when mongod has just been restarted.

  • There's also a TTL index on the collection and Mongo TTL index are most error-prone from my previous experience. Moreover logs never seem to show the TTL index deletion stats.

  • I have two identical (same VM, configuration and mongodb 2.6.4) instances of mongod running on different machines and I'm trying to write 20K documents to both of them at the same time. The document gets committed on one machine in 10 seconds and it has log entries like these:

    command day_raw_tweet_db.$cmd command: insert { $msg: "query not recording (too large)" } keyUpdates:0 numYields:0 locks(micros) w:17 reslen:40 392ms

  • The log entries on the second instance (which is same as before) shows individual documents one per line in log output and as feared it takes around 400 seconds to insert the same number of documents.

  • There's no read load on both the instances and they both are equally idle.

- Can someone please help me as I'm going crazy over this mongodb behaviour.The intermittent bad behaviour is the worst anyone can cope with.

  • 1
    why don't you post this on mongodb-user google group - this isn't a simple question/answer - there is a lot of information that would help here. Without more details about what your application is doing, what the indexes are, how the space is allocated, what sort of storage, OS settings, etc. it would mostly be guessing. Btw, why are you sending 20K documents - they will be split into 1K batches by driver anyway, why not have more threads sending 1K batches - also make sure that you use "unordered" bulk inserts. – Asya Kamsky Sep 13 '14 at 1:04
  • Mongodb extremely slow with bulk inserts and showing erratic behavior <--- that is normal for MongoDB, what's the problem? – Gaius Sep 13 '14 at 8:41
  • @AsyaKamsky Will surely do and put an update here. – VaidAbhishek Sep 15 '14 at 14:56
  • I use TTL indices in several production scenarios and can not confirm that they are unreliable, quite the contrary. Keep in mind though that the documents will be invalid the second they are supposed to, but removed only by the TTLMonitor thread which kicks in every 60 seconds. You can have some poor man stats on TTL in the output of db.serverStatus. Please also let us know which OS, FS and type of disk you are using. – Markus W Mahlberg Sep 18 '14 at 9:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.