Should all machines which run mongo instances (mongod or mongos) in a Mongo DB cluster have unique hostnames (also at OS level)?

Question

I have been using Mongo DB Cluster for some data analysis (Mongo DB version 2.4.5) and while inserting some data into the cluster, I found an error which is the same as mentioned in the bug : https://jira.mongodb.org/browse/SERVER-10578

My configuration is as follows:

1) Node1 - mongoconfig server, mongod shard

2) Node2 - mongoconfig server, mongod shard

3) Node3 - mongoconfig server, mongos

4) Node4 - mongos

I changed the OS hostnames of all the machines and made sure that hostnames specified in the DNS server and OS hostnames are the same. After that, the problem got solved.

But now I have a requirement that I need to add mongos instances (nodes/machines) on the fly which will have different IP addresses but the same hostname (both at the OS level and in DNS entries). I will still continue to have unique hostnames (both at OS level and in DNS entries) for all the mongo config servers and mongo shard servers.

Will that work correctly or will I run across the same problem as mentioned in the bug?

Does the requirement of unique hostnames at OS level hold true even for mongos instances or is it just for shards?

Answer 1

Actually, it is a Very Bad Idea™ to do a load balancing this way with MongoDB. First of all, you can give multiple mongos instances to the driver. If one mongos goes down or becomes unavailable due to overload, the driver will be happy to access the next available mongos. Second, the config servers and mongod instances get unnecessary load for continuously updating the cluster "map". Third, they most likely will initiate and tear down connections continuously, too.

The solution would be to use CNAMEs at the DNS level instead of multiple A records for a host, but maintain unique hostnames on the OS level. That is, the hostname command should return a unique value on every host of the cluster, including the mongos instances.

A word on DNS based round robin load balancing: It's crap when used on it's own. You don't have any control over the load actually sent to each of the balanced servers. In case you have caching DNS servers between you and your biggest group of unique users, you actually might worsen the load for an individual server. You better use a "real" loadbalancer in front of your servers, something like HAProxy, varnish or nginx (depending on your application). In case you need / want to have more than one of those, you can use DNS based round robin load balancing on these, though from my experience, it is better to make one load balancer highly available to prevent a single point of failure.