0

For testing I killed the primary node of my two-node replica set while the cluster was under heavy write load.

I was expecting the secondary node would take over after a few seconds, but this isn't happening.

shard0:SECONDARY> rs.status()
{
    "set" : "shard0",
    "date" : ISODate("2014-10-22T15:11:52Z"),
    "myState" : 2,
    "members" : [
        {
            "_id" : 1,
            "name" : "10.128.42.177:27020",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 10270,
            "optime" : Timestamp(1413990107, 52),
            "optimeDate" : ISODate("2014-10-22T15:01:47Z"),
            "self" : true
        },
        {
            "_id" : 2,
            "name" : "10.128.42.188:27020",
            "health" : 0,
            "state" : 8,
            "stateStr" : "(not reachable/healthy)",
            "uptime" : 0,
            "optime" : Timestamp(1413990103, 49),
            "optimeDate" : ISODate("2014-10-22T15:01:43Z"),
            "lastHeartbeat" : ISODate("2014-10-22T15:11:50Z"),
            "lastHeartbeatRecv" : ISODate("2014-10-22T15:01:43Z"),
            "pingMs" : 0,
            "syncingTo" : "10.128.42.177:27020"
        }
    ],
    "ok" : 1
}

As you can see at the time of printing rs status the last heartbeat received was already 10 minutes old.

What am I doing wrong?

4

You are missing a node, either another data bearing node or an arbiter.

Here is why. A replica set needs a quorum > 50% of the original replica set members. Additionally, if a single node could decide to become primary, every network partitioning, every failing switch would result in a split brain situation, where two nodes assume they are primary.

Add an arbiter. They are cheap in terms of CPU, RAM and disk space, so you can run it on a small VM. Run it with

mongod --replSet shard0 --smallfiles --nojournal --noprealloc 

which further decreases the need of space.

Do NOT run the arbiter on one of the other nodes - it would leverage it's purpose.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.