5

I have a collection in MongoDB with 800M+ records. I wanted to do some cleanup, but the remove operation is really really (unusably) slow - even deleting 1000 records can take 10+ minutes.

The structure of one record in the collection is something like this:

{
     'stID' : '00000001',
     'recordID' : '28-4',
     'date' : ISODate('2017-06-06 20:00:00'),
     ......
}

There is a compound index on {'stID' : 1, 'recordID' : 1, 'date' : 1} and my delete query is like this: db.coll.remove({ 'stID' : '00000001', 'recordID' : '28-4' }, { multi: true });

The database is part of a replicaSet, with one secondary and one arbiter, and I do understand that the delete operation has to fill the oplog with the _ids of each record to delete, but still I find it hard to understand why this could be so slow, especially since:

  • there is no considerable system or IO activity on the primary or the secondary
  • the secondary is not lagging behind the primary

We have never experienced problems with find or write/update operations. Can you suggest what other parameters/commands/tools should I use to see what is going on with the query and the servers.

The storage engine we use is WiredTiger. The server has 128GB of RAM and fast SSD drives in a RAID 1 configuration.

  • What is your MongoDB version? – kevinadi Jun 28 '17 at 23:56
  • Would you be able to share a log message for one of the remove operations (docs.mongodb.com/manual/reference/log-messages). The log messages for slow operations typically contain a good deal of information about how an operation was was performed. – Adam Harrison Jun 29 '17 at 0:20
  • What does explain output say? Have you examined? – harshavmb Jun 29 '17 at 5:06
  • Please, copy here what command db.coll.remove({ 'stID' : '00000001', 'recordID' : '28-4' }, { multi: true }).explain() says, maybe then we managed to give answer to your question. – JJussi Jul 4 '17 at 4:09
0

I've had improved performance by using bulkWrite:

db.ig_posts.bulkWrite([
   { deleteMany: { "filter": { /* your query here */ } }
]);

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.