1

My cleanup orphans on a collection is running for 3 days now. Log shows a few million records being deleted once in a while. However, most of the time nothing is deleted.

I SHARDING [conn3919] rangeDeleter deleted 0 documents for < my collection name >

Not sure why this is so slow. Is this expected? We do have over 100 shards. When we tried to run cleanup on all shards in parallel, we noticed that all cleanups were stuck for a day (no cleanup logs). So we had to run it sequentially. As an alternate to cleanupOrphaned script, we tried adding a new replica set (after shutting down secondary nodes) thinking that it will sync from primary without orphans. Sync is always very fast comparatively. However, after sync the new replica set also had orphans! So that didn't work.

Any advice on a quicker way to cleanup orphans is highly appreciated. We are using Mongo v3.4

Also, is it possible to have orphans when balancer is off? We had our balancer off always but still ended up having orphans.

  • 'orphans' are just documents like other ones, so yes, secondary nodes will replicate those from primary. Consept 'orphan' is just sharding related and means that during balancing, original documents were not deleted from original node. Documents are 'orphans' because chunk range record points now to "new location". There is some kind of problem at the system if you have lot's of orphans. Reason why orphan remove takes lot's of time is that, process reads thru all documents and check (at every document) that "do this document belongs to this shard, if not, it's orphan. – JJussi Jun 13 '18 at 13:36
  • Thanks @JJussi. I understand this but what I don;t understand are two things. 1) Primaries know how to hide orphans, however secondaries for some reason can not do this. 2) Since primaries know how to hide orphans, I thought the new replica sets synced from primary should be able to hide orphans as well. Also, I am wondering why can't I run the cleanupOrphans concurrently on multiple shards. – Bijith Kumar Jun 13 '18 at 17:48
  • Primaries don't hide orphans, config server with list of chunks just point clients to right shards (and not to the wrong ones). If you connect directly to the shard (replica set) and query those orphan documents, they are there. Those documents are there because balancing process is just "copy documents to new shard and then delete from original location", if that delete don't happen, documents are still there, even chunk-collection already points to new location. – JJussi Jun 14 '18 at 19:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.