Why is my 4-node replicaset setup is not working?

Question

I have a 4 Node setup which is being hosted On-Prem by 2 Host Servers (DC1, DC2)

192.168.0.1 - Primary (Located in DC1)
192.168.0.2 - Secondary (Located in DC2)
192.168.0.3 - Arbiter (Located in DC2)
192.168.0.4 - Arbiter (Located in DC1)

We opted for this setup since we do perform maintenance on our servers from time to time which may require restarting/shutting down the Main Server located in either DC1/DC2.

Since our previous setup is only 3 Nodes

1 Primary (DC1)
1 Secondary (DC2)
1 Arbiter (DC2)

This have a single point of failure that is relying on DC2. If we shutdown DC2, the Primary will be the one remaining and since there are only 1 node. The election of the primary would not take effect since it needs at least 2 voting members.

However upon adding another arbiter on DC1. When DC2 is down, there will be 2 Nodes remaining (Primary and the 2nd Arbiter on DC1). I expected that the replicaset will still be available. But then the Primary became a Secondary and no Primary is being elected. Does it need to be a majority voting and odd numbers of nodes are the suggested setup? Is there a way to tweak that settings?

Answer 1

2 out of 4 is not the majority, thus a PRIMARY cannot be elected. Having more than one ARBITER is not recommended and in newer releases, it is even not permitted by default, see Concerns with Multiple Arbiters

For maintenance work (which is planned) you can reconfigure the member to a passive member, i.e. set { votes: 0, priority: 0 } the you can restart/shutdown the server without interruption of service. After your maintenance revert the config.

See also https://stackoverflow.com/questions/69658590/mongoclient-to-connect-to-multiple-hosts-to-handle-failover/69666511#69666511