3

I'm baffled by the fact that this isn't addressed in the official docs (or at my inability to track it down). From the 3.4 docs:

Only add an arbiter to sets with even numbers of voting members. If you add an arbiter to a set with an odd number of voting members, the set may suffer from tied elections.

So we have two possible scenarios:

  1. A replica set with an odd number of data nodes, which requires no arbiter (and in fact, adding an arbiter can be detrimental per above)
  2. A replica set with an even number of data nodes, and an arbiter

This is great so long as things are running smoothly. But what if a data node goes down in each of these scenarios?

  1. We now have an even number of nodes with no arbiter. As I understand it this means you cannot have a successful election.
  2. We now have an odd number of nodes, plus an arbiter. Above docs state that this can lead to tied elections.

This is so painfully obvious, why can't I find any discussion or documentation on it? But what I really want to know is, how do I correctly handle these scenarios?

2

The context you are missing is that failure of a replica member does not affect the number of configured members the replica set has (or the required voting majority to maintain a primary). Any changes in fault tolerance or replica set election requirements will involve a re-configuration of the replica set (eg. adding/removing members or changing voting members) rather than a replica set member state change (eg. DOWN, RECOVERING, ..).

A three node replica set with one down member still has three members (with a strict majority of two). If you think of it as analogous to a RAID configuration, a three node replica set with one member down is running in degraded mode and cannot tolerate the failure of any further voting members. This allows for continued availability, but you will want to recover or replace the unavailable member to return the replica set to a healthy state.

You can check the current replica set configuration with rs.conf() and the current state of members with rs.status().

  • But if there are e.g. 3 data nodes (no arbiter) and the primary goes down, how can a successful election be held? "failure of a replica member does not affect the number of configured members the replica set has" -- but clearly the "down" node can't participate in the election. So I don't think I understand the point you're making there. – Madbreaks Feb 22 '17 at 0:07
  • @Madbreaks If you have 3 voting nodes in the replica set configuration and any single node is unavailable, the remaining 2 nodes still represent a strict majority of the replica set configuration (i.e. 2/3 nodes) and can elect a primary. The primary election requires a strict majority of voting nodes, so either 2 or 3 votes will elect a primary. With an even number of voting nodes (for example, 4) a strict majority will require n/2+1 votes (so 3 votes). With all members healthy, a 4 node replica set with an even number of votes could result in a 2/2 split and take longer to reach consensus. – Stennie Feb 22 '17 at 2:11
  • @Madbreaks The general point is that properly deployed replica sets can handle unavailable members without any reconfiguration. Arbiters do not play a special role in the determination of a primary, and it is expected that some number of members (up to the fault tolerance of the replica set configuration) may be unavailable. For example, this feature can be used to minimise downtime for rolling maintenance/upgrades. – Stennie Feb 22 '17 at 2:18
  • "If you have 3 voting nodes in the replica set configuration and any single node is unavailable, the remaining 2 nodes still represent a strict majority of the replica set" --- ahhhh, I see! That makes sense. Thanks much for your insight. – Madbreaks Feb 22 '17 at 18:16
1

Arbiters are used to help you enforce whatever method you are using to maintain your uptime and help have a node that doesn't take in data nor processes any data requests, but purely acts as a voter. This could be useful if you have an even number of powerful machines but need another voter in the cluster.

If you do lose a data node, you should replace it ASAP to fit your SLA regardless. However if you want to protect yourself from going down due to a single machine failing, you could add several very tiny small arbiters into the cluster. This way you will need multiple machines to go down. Arbiters are very simple and as long as they maintain network communication they work well.

Arbiters could also be useful for geolocation solutions, such as having a certain amount of arbiters in a different region which will let you failover to different data centers.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.