1

I have the following working replica set:

{
    "_id" : "repl1",
    "version" : 6,
    "members" : [
            {
                    "_id" : 0,
                    "host" : "hostA",
                    "priority" : 10
            },
            {
                    "_id" : 3,
                    "host" : "hostB",
                    "priority" : 8
            },
            {
                    "_id" : 4,
                    "host" : "hostC",
                    "priority" : 6
            }
    ]
}

HostA is the primary, hosts B and C are secondary. There's no arbiter.

I'm testing a failover scenario by shutting down hostA and waiting 10 minutes.

I would expect hostB to be elected to primary, but nothing happens. Hosts B and C are still secondary and my application fails to connect to the replica set.

All hosts are on version 2.6.7 of MongoDB.

Is there anything I could be doing wrong?

Edit:

rs.status() before shutting down host A:

{
    "set" : “repl1”,
    "date" : ISODate("2015-02-17T03:11:00Z"),
    "myState" : 2,
    "syncingTo" : “hostA”,
    "members" : [
            {
                    "_id" : 0,
                    "name" : “hostA”,
                    "health" : 1,
                    "state" : 1,
                    "stateStr" : "PRIMARY",
                    "uptime" : 44244,
                    "optime" : Timestamp(1424142650, 1),
                    "optimeDate" : ISODate("2015-02-17T03:10:50Z"),
                    "lastHeartbeat" : ISODate("2015-02-17T03:10:59Z"),
                    "lastHeartbeatRecv" : ISODate("2015-02-17T03:10:59Z"),
                    "pingMs" : 1,
                    "electionTime" : Timestamp(1424098422, 1),
                    "electionDate" : ISODate("2015-02-16T14:53:42Z")
            },
            {
                    "_id" : 3,
                    "name" : “hostB”,
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 119566,
                    "optime" : Timestamp(1424142650, 1),
                    "optimeDate" : ISODate("2015-02-17T03:10:50Z"),
                    "self" : true
            },
            {
                    "_id" : 4,
                    "name" : “hostC”,
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 44852,
                    "optime" : Timestamp(1424142650, 1),
                    "optimeDate" : ISODate("2015-02-17T03:10:50Z"),
                    "lastHeartbeat" : ISODate("2015-02-17T03:10:58Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 0,
                    "syncingTo" : “hostA”
            }
    ],
    "ok" : 1
}

rs.status() after shutting down host A:

{
    "set" : "repl1",
    "date" : ISODate("2015-02-17T06:18:05Z"),
    "myState" : 2,
    "members" : [
            {
                    "_id" : 0,
                    "name" : "hostA",
                    "health" : 0,
                    "state" : 8,
                    "stateStr" : "(not reachable/healthy)",
                    "uptime" : 0,
                    "optime" : Timestamp(1424151278, 2),
                    "optimeDate" : ISODate("2015-02-17T05:34:38Z"),
                    "lastHeartbeat" : ISODate("2015-02-17T06:17:53Z"),
                    "lastHeartbeatRecv" : ISODate("2015-02-17T06:17:22Z"),
                    "pingMs" : 0
            },
            {
                    "_id" : 3,
                    "name" : "hostB",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 130791,
                    "optime" : Timestamp(1424151278, 2),
                    "optimeDate" : ISODate("2015-02-17T05:34:38Z"),
                    "infoMessage" : "syncing to: hostA",
                    "self" : true
            },
            {
                    "_id" : 4,
                    "name" : "hostC",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 56077,
                    "optime" : Timestamp(1424151278, 2),
                    "optimeDate" : ISODate("2015-02-17T05:34:38Z"),
                    "lastHeartbeat" : ISODate("2015-02-17T06:18:04Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 1,
                    "lastHeartbeatMessage" : "syncing to: hostA"
            }
    ],
    "ok" : 1
}

After rs.stepDown():

{
    "set" : "repl1",
    "date" : ISODate("2015-02-17T06:23:20Z"),
    "myState" : 2,
    "syncingTo" : "hostA",
    "members" : [
            {
                    "_id" : 0,
                    "name" : "hostA",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 304,
                    "optime" : Timestamp(1424151278, 2),
                    "optimeDate" : ISODate("2015-02-17T05:34:38Z"),
                    "lastHeartbeat" : ISODate("2015-02-17T06:23:19Z"),
                    "lastHeartbeatRecv" : ISODate("2015-02-17T06:23:19Z"),
                    "pingMs" : 1
            },
            {
                    "_id" : 3,
                    "name" : "hostB",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 131106,
                    "optime" : Timestamp(1424151278, 2),
                    "optimeDate" : ISODate("2015-02-17T05:34:38Z"),
                    "infoMessage" : "syncing to: hostA",
                    "self" : true
            },
            {
                    "_id" : 4,
                    "name" : "hostC",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 56392,
                    "optime" : Timestamp(1424151278, 2),
                    "optimeDate" : ISODate("2015-02-17T05:34:38Z"),
                    "lastHeartbeat" : ISODate("2015-02-17T06:23:19Z"),
                    "lastHeartbeatRecv" : ISODate("1970-01-01T00:00:00Z"),
                    "pingMs" : 1,
                    "lastHeartbeatMessage" : "syncing to: hostA",
                    "syncingTo" : "hostA"
            }
    ],
    "ok" : 1
}
  • Please share: 1) rs.status() before and after the shutdown. 2) the logs from hostB and hostC after the shutdown occurs 3) issue an rs.stepdown() on A and share the results – Antonios Feb 16 '15 at 20:27
  • There is nothing obviously wrong with the config here, so time to look at the logs. The logs will tell you why there is no primary being elected. Given the priorities I would expect Host B to be elected, so that's the first log I would look at. Similarly the logs from Host A after the step down would be instructive. If you post the relevant sections, happy to take a look. – Adam C Feb 17 '15 at 13:19
  • 2
    ah ha! The logs said that hostC didn't think hostB was electable and it turns out I had forgotten to add hostC IP address to hostB firewall. I haven't tested it yet because these are production servers, but will test it late at night when traffic is low. I'm pretty sure it will work now. Thank you! – perfect_element Feb 17 '15 at 14:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.