I have a replica set with 2 members and 1 arbiter. The data center hosting the Secondary was having intermittent network issues today, which caused the Secondary to go down several times, often up/down a few times within a minute.
To my surprise, at a certain point this caused the Primary to go to ROLLBACK state bringing the whole replica down, even though the Primary and Arbiter had no connectivity issues.
The ROLLBACK state persisted for several minutes, until I restarted the DB service and removed the faulty Secondary from the replica set.
What caused this behavior and how can I prevent this from happening?
If I replace the Arbiter with another member, will this make things more reliable?