We have a sharded cluster using replica sets with local secondaries (pingMs = 0), and remote secondaries (pingMs = 40). So, each replica set has a local secondary and a remote secondary. Chunk migration used to choose the local secondary when moving a chunk to satisfy its "majority" writeconcern used by Mongo's chunk migration. And our transfer speed was reasonably fast.

Today we added another shard using a replica set configured identically to the other shards. And, now when it does chunk migration to that new shard, it's using the remote secondary instead of the local secondary to satisfy the "majority" writeconcern and transfers are very slow due the network latency?? Am I correct in thinking that Mongo should be choosing the local secondary to satisfy the "majority" writeconcern because it has a much lower ping time?

I've compared rs.config() between the new shard and one of the older shards and they are identically configured. We're using MongoDB 3.2.6.

  • And after about 12 hours of doing this, it decided to start using the local secondary, and chunk migration transfer rates went back to normal! – Jim Reitz Apr 5 '17 at 21:36

Actually mongodb don't "choose" neither of those secondaries. Write concern majority is fulfilled when first secondary has replicated that change successfully and reported that information back to the primary. In this situation, somehow your "local" secondary was lagging with opLog replication, reason for that may or may not be found if you analyze that nodes mongod.log file.

| improve this answer | |
  • Unfortunately, I guess it's too late to check the logs for that. But I'll keep that in my back pocket in case it ever happens again. Thanks! – Jim Reitz Apr 13 '17 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.