Primary replica set server goes secondary after secondary fails

Question

I have a 2 servers replica set that, after the secondary fails the primary goes into secondary mode while the secondary is in STARTUP2 (recovering). The problem with this is that I can't use the collection stored in that replica set freely, I'm getting errors trying to use the collection:

pymongo.errors.OperationFailure: database error: ReplicaSetMonitor no master found for set: rs2

Sometimes if I restart the mongod instances, the server rs2-1 is the primary for a while, but after some time (while the secondary is recovering) I see this in the logs of rs2-1 (the primary):

Tue May  7 17:43:40.677 [rsHealthPoll] replSet member XXX.XXX.XXX.XXX:27017 is now in state DOWN
Tue May  7 17:43:40.677 [rsMgr] can't see a majority of the set, relinquishing primary
Tue May  7 17:43:40.682 [rsMgr] replSet relinquishing primary state
Tue May  7 17:43:40.682 [rsMgr] replSet SECONDARY
Tue May  7 17:43:40.682 [rsMgr] replSet closing client sockets after relinquishing primary

Is there an easy way to make the primary keep being primary after the secondary fails? Am I doing something wrong?

Thanks in advance!

Answer 1

MongoDB requires a primary to be able to see a majority of the set in order to remain a primary. The majority is defined as being greater than n/2 machines where n is the number of machines in the replica set. So, if n is 1, then one machine constitutes a majority. When n is 2 or 3, 2 machines constitute a majority.

When the secondary is added, the primary now requires to be able to access two machines, itself and the secondary. When the newly added secondary goes down, the primary no longer sees a majority, and steps down.

This scenario is why an arbiter is recommended in addition to the primary and secondary. If you have two machines and one arbiter in the replica set, assuming the arbiter always stays up (which should be true because the arbiter does barely anything), then as long as one other machine is up, the arbiter will ensure that a primary will exist, because the other machine and the arbiter constitute a majority (2 out of 3).

Answer 2

I have find another way but I dont know if it is the best solution. Imp pretty sure about what member must be the primary, thus you can alter the weight of this member in order to force that it will be the primary instead the secondary fails.

Thats my solution:

    rs0:PRIMARY> rs.status()
{
    "set" : "rs0",
    "date" : ISODate("2018-04-04T10:00:21Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "mongodb01:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 64052,
            "optime" : Timestamp(1522835502, 1),
            "optimeDate" : ISODate("2018-04-04T09:51:42Z"),
            "electionTime" : Timestamp(1522835358, 1),
            "electionDate" : ISODate("2018-04-04T09:49:18Z"),
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "mongodb02:27017",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(0, 0),
            "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
            "lastHeartbeat" : ISODate("2018-04-04T10:00:19Z"),
            "lastHeartbeatRecv" : ISODate("2018-04-04T09:54:38Z"),
            "pingMs" : 0
        }
    ],
    "ok" : 1
}

For achieve this read https://docs.mongodb.com/manual/tutorial/configure-a-non-voting-replica-set-member/

I have done;

cfg = rs.conf();
cfg.members[0].votes = 2;
rs.reconfig(cfg);

You can check:

rs0:PRIMARY> rs.config()
{
    "_id" : "rs0",
    "version" : 5,
    "members" : [
        {
            "_id" : 0,
            "host" : "mongodb01:27017",
            "votes" : 2
        },
        {
            "_id" : 1,
            "host" : "mongodb02:27017"
        }
    ]
}

NOTE: You'll need to do this operations in a primary member.

Answer 3

Note While not recommended, the minimum supported configuration for replica sets includes one primary, one secondary, and one arbiter. The arbiter requires fewer resources and lowers costs but sacrifices operational flexibility and redundancy.

from http://docs.mongodb.org/manual/core/replica-set-architectures/

you don't have enough members to maintain quorum