I have a MongoDB replica set with 3 nodes and 1 arbiter.
The nodes have priority 0, 1, and 10. The arbiter has priority 1. They all have 1 vote.
The priority 10 was PRIMARY, while the priority 0 and 1 were both SECONDARY.
I set the default write concern to
As a test I cleanly shut off one of the SECONDARY, namely the priority 0 node, with
What I expected to happen is that everything continued to operate normally. Since the primary was still up, and the write concern was 1 (meaning only the primary needs to acknowledge), then everything should still be ok.
However, what I observed is that sometimes a write query would work, while other times it would block forever. My system as a whole stopped working. As soon as I brought the secondary back up, it started working again normally.
Why is this happening? One of my co-workers said it's because the queries try to write to any one of the nodes, and if they pick the secondary that is down then it will fail, but that isn't my understanding from what the documentation says.
Is the replica set really supposed to fail if a SECONDARY goes down? Isn't the point of the replica set to solve precisely this issue?
Additional info: when I set the priority 0 node to also have 0 votes, shutting it down didn't affect anything. So it seems it having a vote contributes to the issue... but not sure what it could be.