How can MongoDB elect a new primary when a datacenter has failed?

Question

I am curious about the following situation when setting up a MongoDB cluster.

2 Datacenters, 2 nodes in each data center for a total of a 4 node replicaset. One of the two datacenters has become unavailable for any reason, thus blacking out the primary and one of the secondaries.

Per fault tolerance section of https://docs.mongodb.com/manual/core/replica-set-architectures/: I need at minimum 3 votes to elect a new primary, but will only have 2 surviving voters.

I am unable to extend into a 3rd datacenter.

How can I setup the cluster to be able to withstand a datacenter failure and continue to write in the surviving datacenter?

Answer 1

You need another replica set member (or arbiter) in a 3rd data center. But if you cannot extend to a 3rd center, then add the 5th member of the replica set to your main data center. Your backup data center will remain with two members.

If backup data center goes down, main center will stay up and replica set operations will continue.

If main center goes down, replica set operations will be read only and you will need to perform manual intervention.

References:

Deploy a Geographically Redundant Replica Set

Replica Sets Distributed Across Two or More Data Centers

Answer 2

In this scenario, it would be required to remove the lost nodes' ability to vote from the server configuration of the remaining nodes. This shrinks the effective cluster size to 2, where a priority system can create a quorum and elect a primary.

Example:

#Note: members[0],[1] are in datacenter1, members[2],[3] are in datacenter2. 
cfg = rs.conf()
cfg.members[0].priority = 0
cfg.members[1].priority = 0
cfg.members[0].votes = 0
cfg.members[1].votes = 0
rs.reconfig(cfg)
#Now members[2] is elected primary.

Unfortunately, this takes manual intervention at the time of data center failure to restore service to anything relying on writing to this database. It also requires manual intervention to restore the cluster to the starting state after the lost nodes return to service and have recovered (to give them voting privileges back).