4

I'm working with a MongoDB collection containing over 250 million documents that has an existing TTL index and is written to with a pretty high frequency. I'd like to reduce the TTL value for the index from 60 days to 30 days, while minimizing production impact. It seems that I have two options: dropping and recreating the index in the background or running the collMod command to modify the index. The simplicity of the collMod command is appealing, but I'm concerned about the impact it might have, especially since there appears to be no way to run it in the background. Would dropping/recreating the index lead to less lock contention and have less of an impact?

  • I ended up going the route of dropping and recreating the index, since I knew that it would be run in the background. It took ~4 hours, but seemed to work fine. I was a bit unhappy to see the level of load introduced by the reaper process that cleans up the documents older than the TTL, but it doesn't seem to be preventing other operations. – Matt Passell Dec 16 '15 at 21:33
3

The first you need to do is calculate how many documents the index will delete if you change the 60 days into 30. That is a simple count with {TTL_FIELD: $gt:30days}.

If the number of documents is big (how "big" depends from the server specs and the workload) I recommend to run an incremental Colmod like (pseudocode):

While achieve 30 days {

NUM = 60 days;

NUM = NUM - threshold;

db.runCommand( {"collMod" : "COLLLECTION" , "index" : { "keyPattern" : {TTL_FIELD : 1} , "expireAfterSeconds" : NUM } } )

wait 2 minutes;}

If the number of deleted document is small just do the collMod at once or on 2-3 steps manually.

Your approach to drop and recreate the index takes more time and if you have lot of documents for deletion you will get performance degradation when the index completes.

Hope that helps

  • Is it your impression that the collMod just updates the expireAfterSeconds value and then lets the TTL (reaper) thread discover that it might have more documents to delete? If so, the change to the index itself would be very fast. In my case, each expired day has about ~4m documents and (probably because the collection is being actively written to at the same time) takes about 2 hrs to remove. So, 30 days worth of docs translates to 60 hours. Ouch... – Matt Passell Dec 18 '15 at 16:11
  • 1
    Yes this is exactly how the collMod works and its easy to re-produce it on a test mongo instance. The change is fast but when the TTL deletion thread executes (once per minute) it will delete all matching documents. The question to you have to answer in these cases is: Can my app handle the deletion of all documents at once? If yes fine change the expireAfterSeconds if not use the script to run the deletion in steps. – Antonios Dec 21 '15 at 21:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.