MongoDB replica set node failure without arbiter

Question

I'm baffled by the fact that this isn't addressed in the official docs (or at my inability to track it down). From the 3.4 docs:

Only add an arbiter to sets with even numbers of voting members. If you add an arbiter to a set with an odd number of voting members, the set may suffer from tied elections.

So we have two possible scenarios:

1. A replica set with an odd number of data nodes, which requires no arbiter (and in fact, adding an arbiter can be detrimental per above)
2. A replica set with an even number of data nodes, and an arbiter

This is great so long as things are running smoothly. But what if a data node goes down in each of these scenarios?

1. We now have an even number of nodes with no arbiter. As I understand it this means you cannot have a successful election.
2. We now have an odd number of nodes, plus an arbiter. Above docs state that this can lead to tied elections.

This is so painfully obvious, why can't I find any discussion or documentation on it? But what I really want to know is, how do I correctly handle these scenarios?

Answer 1

The context you are missing is that failure of a replica member does not affect the number of configured members the replica set has (or the required voting majority to maintain a primary). Any changes in fault tolerance or replica set election requirements will involve a re-configuration of the replica set (eg. adding/removing members or changing voting members) rather than a replica set member state change (eg. DOWN, RECOVERING, ..).

A three node replica set with one down member still has three members (with a strict majority of two). If you think of it as analogous to a RAID configuration, a three node replica set with one member down is running in degraded mode and cannot tolerate the failure of any further voting members. This allows for continued availability, but you will want to recover or replace the unavailable member to return the replica set to a healthy state.

You can check the current replica set configuration with rs.conf() and the current state of members with rs.status().

Answer 2

Arbiters are used to help you enforce whatever method you are using to maintain your uptime and help have a node that doesn't take in data nor processes any data requests, but purely acts as a voter. This could be useful if you have an even number of powerful machines but need another voter in the cluster.

If you do lose a data node, you should replace it ASAP to fit your SLA regardless. However if you want to protect yourself from going down due to a single machine failing, you could add several very tiny small arbiters into the cluster. This way you will need multiple machines to go down. Arbiters are very simple and as long as they maintain network communication they work well.

Arbiters could also be useful for geolocation solutions, such as having a certain amount of arbiters in a different region which will let you failover to different data centers.