My cleanup orphans on a collection is running for 3 days now. Log shows a few million records being deleted once in a while. However, most of the time nothing is deleted.
I SHARDING [conn3919] rangeDeleter deleted 0 documents for < my collection name >
Not sure why this is so slow. Is this expected? We do have over 100 shards. When we tried to run cleanup on all shards in parallel, we noticed that all cleanups were stuck for a day (no cleanup logs). So we had to run it sequentially. As an alternate to
cleanupOrphaned script, we tried adding a new replica set (after shutting down secondary nodes) thinking that it will sync from primary without orphans. Sync is always very fast comparatively. However, after sync the new replica set also had orphans! So that didn't work.
Any advice on a quicker way to cleanup orphans is highly appreciated. We are using Mongo v3.4
Also, is it possible to have orphans when balancer is off? We had our balancer off always but still ended up having orphans.