If I have six nodes (3 node in each DC), would I need an arbiter?
Yes - as Mongo docs mentioned an odd number is ideal.
If you have 6 Nodes and 1 goes down, the remaining 5 will elect a new primary. When 2 go down, 4 will elect a primary. When 3 do go down, you only have 3 left which are no longer a majority and all 3 will become read-only until a 4th node comes back online.
If you have 6 + an arbiter, then the scenario plays out slightly more beneficially. 1 node goes down, 5+1 elect a new primary. 2 go down, 4+1 elect a new primary. 3 go down, 3+1 still has a majority (3+1 / 7) so they can elect a new primary. Finally, when 4 nodes are down you are left with 2 nodes up and 1 arbiter, which is less than the majority (2+1 / 7) so you'll enter into read-only until a new node comes back online.
An odd number gives you one more level of outage before you lose the majority of your voting nodes and an arbiter is a cheap way to gain that benefit.
If I have five nodes (3 in DC1 and 2 in DC2), if DC1 fails, wouldn't I
need an arbiter in DC2?
In this config, adding an arbiter would give you an even number, so that's not ideal (see above). However, if DC1 goes down you're also losing your majority automatically and thus down to read-only by default. Although it might seem strange, this is the recommended setup from Mongo.
But why?
You could have 3 in DC1 and 3 in DC2 with an arbiter in DC2, which would allow you to maintain a voting majority in DC2 if DC1 were to completely fail. This works if you want automatic failover into DC2. But if DC2 goes down and a node in DC1 goes down, you also lose DC1 majority and end up with read-only nodes - probably not ideal.
That's why Mongo recommends that you set the priority for nodes in DC2 to 0. In this case you should have 3 nodes + 1 arbiter in DC1 and 3 nodes in DC2. You will need to manually fail over to DC2 if DC1 completely fails, but it gives you the best case scenario as far as keeping DC1 active and primary.
