0

I have a MongoDB replica set with 3 nodes and 1 arbiter.

The nodes have priority 0, 1, and 10. The arbiter has priority 1. They all have 1 vote.

The priority 10 was PRIMARY, while the priority 0 and 1 were both SECONDARY.

I set the default write concern to {w: 1}.

As a test I cleanly shut off one of the SECONDARY, namely the priority 0 node, with db.shutdownServer().

What I expected to happen is that everything continued to operate normally. Since the primary was still up, and the write concern was 1 (meaning only the primary needs to acknowledge), then everything should still be ok.

However, what I observed is that sometimes a write query would work, while other times it would block forever. My system as a whole stopped working. As soon as I brought the secondary back up, it started working again normally.

Why is this happening? One of my co-workers said it's because the queries try to write to any one of the nodes, and if they pick the secondary that is down then it will fail, but that isn't my understanding from what the documentation says.

Is the replica set really supposed to fail if a SECONDARY goes down? Isn't the point of the replica set to solve precisely this issue?

Additional info: when I set the priority 0 node to also have 0 votes, shutting it down didn't affect anything. So it seems it having a vote contributes to the issue... but not sure what it could be.

Improve this question

2 Answers 2

Reset to default
1

Yes, basic problem here is even count of votes! You never should have even an count of votes. Arbiter should be used ONLY when there is even count of data bearing nodes, like Primary + one (or three) secondary. So, remove that arbiter because you have already replica set with three nodes (and votes).

Improve this answer
4
  • 1
    Do you really think this causes the problem? Node 1 is successfully elected as PRIMARY, so number of votes should not be the issue. Of course, having 4 voting members is a poor setup but it should work if one SECONDARY goes offline.
    – Wernfried Domscheit
    Commented Dec 9, 2022 at 8:38
  • 1
    Normally it should work, but going through (all nodes) logs will tell the truth. I guess that there is a fight between nodes.
    – JJussi
    Commented Dec 10, 2022 at 18:55
  • I asked for the same information already at stackoverflow.com/questions/74729135/… - so far no response.
    – Wernfried Domscheit
    Commented Dec 10, 2022 at 18:58
  • @WernfriedDomscheit The arbiter should definitely be removed in this scenario. Since there are 4 voting members configured, majority writes will require acknowledgement from 3/4 members. Since there are only 3 members that can ack writes (an arbiter only votes), having a secondary down will affect majority commits and lead to memory pressure similar to Mitigate Performance Issues with PSA Replica Set.
    – Stennie
    Commented Dec 15, 2022 at 1:59
0

@Stennie provided the correct hint. You must remove the ARBITER node rs.remove(...), then it will work.

Command rs.status() returns this data

majorityVoteCount: 3,
writeMajorityCount: 3,
votingMembersCount: 4,
writableVotingMembersCount: 3,

writeMajorityCount: 3 means, if you have a command with { writeConcern: { w: "majority" } } (and as stated already, for some commands w: "majority" is set automatically) then you must have 3 writing nodes available - the ARBITER is a non-writing member.

When you have PRIMARY and 2 SECONDARY nodes then rs.status() returns

majorityVoteCount: 2,
writeMajorityCount: 2,
votingMembersCount: 3,
writableVotingMembersCount: 3,

Which means for w: "majority" you need only 2 writing nodes available.

Improve this answer
3
  • This makes sense, except I had already set the default write concern to w, and I did a test query from a mongosh session connected to the replica set. Wouldn't this use the default and therefore not have come across this issue?
    – Claudiu
    Commented Dec 18, 2022 at 20:44
  • What do you mean by "not working"? See my update,
    – Wernfried Domscheit
    Commented Dec 19, 2022 at 8:09
  • So the default write concern was {w: 1}. i would expect that this would mean any query which writes, would return as long as one writing member is available. What happened though is that when I made a query that inserted with only 2 of 3 writing nodes available, the query would block and not return. Eventually (once 3rd writing node was up) the query update went through. So it seems it was still waiting for 2 confirms even though the write concern was set to only wait for 1.
    – Claudiu
    Commented Dec 19, 2022 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.