What's the recommended way to deploy config servers?

Assume, we have two shards, each one is a 3-nodes replica set. If I deploy all config servers together with one of these shards, a single network partition can make the second shard completely not functional.

If I deploy 3 config servers in a distributed fashion, then after a network partition both shards will be available, but in read-only mode.

If I deploy config servers to a separate datacenter, it will just move the point of failure from shard-to-shard network part to shard-to-datacenter network part.

Any better options?

  • Mongodb is CP, not AP. If primary for a shard is unreachable, you can't write to it, no matter how you juggle the config servers. – Sergio Tulentsev Aug 31 '15 at 11:11
  • Can you share your planning numbers? Will you have a dozen or 100 shards? What volume of writes are you expecting on the config servers? They are typically extremely lightwight and just store metadata on cluster configuration. Any reliable slow server can work for config. – bmike Aug 31 '15 at 17:27

If you lose one of the three config servers, you just halt migrations, so in a balanced system where you're not exceeding 80% capacity, you should have hours and hours of time to page the duty DBA and bring back that one server.

You could virtualize them or place them on management band / highly reliable servers if you really can't afford to have down time for the cluster to perform chunk migration over a long weekend or other situation where you're not attentive to the config servers.

Most of your planning will be to make sure you understand the network partition scenarios and that the primary stays running and you have enough secondaries to serve read loads as needed. It's hard to generalize other than:

  • Add arbiters to keep the voting members an odd number.
  • Test your outage scenarios
  • Scale the system so you can actually perform rolling outages and verify that things work well without killing your application in practice. You might not test at peak load, but you should be able to take rolling outages routinely several times a week until you're confident in your tools, scripts and configuration.
| improve this answer | |
  • 1
    I'd like to add that you can't add additional shards or shard collections as long as one of the config servers is down, too. – Markus W Mahlberg Sep 7 '15 at 10:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.