MongoDB automatic failover between 2 data centers

Question

If I deploy a MongoDb replica set across only 2 data centers, if one entire data center fails, can there be automatic failover to the other data center?

Consider the situation where 1 data center has a primary, secondary and an arbiter, and the second data center has 2 secondary members. Essentially, a 5 member replica set.

If data center 2 goes down, there will still be 3 members in the replica set in data center 1. That is great.

In reverse, if data center 2 remains up and data center 1 goes down, there will only be 2 members left out of the 5 member replica set. That is not enough to have either secondary vote themselves to become primary.

Is that correct? If not, what am I doing wrong.

If this isn't possible, please let me know so I can explore other options.

Adding a third data center to house the arbiter is not an option.

Answer 1

First, you are correct. If the replica set doesn't have a quorum, it will relegate to secondary state.

The solution is pretty easy: put two data bearing nodes in each of the two datacenters and an arbiter on a cheapo VM in a third datacenter. Hell, a free AWS instance should do the trick. It is very unlikely that you'll loose two of three datacenters at the same time.

From my experience, everything else will fire back on you.

Answer 2

Adding more to Markus's response, in the given configuration of 5 hosts (3 plus 2), and in the event of network partition, data centre 1 is not reachable, you may also plan a contingency of manual execution (since quorum is not obeyed in this case) of forcing a member to become primary - of course you stand to lose data in such intervention.