0

I have 8 instances running on Scaleway , 5 for Data and 3 Arbiter.

And the versions of mongo on them are such :

mongodb2  (mongodb 3.4.9)
mongodb4  (mongodb 3.4.15)
mongodb5  (mongodb 3.4.15)
mongodb6  (mongodb 3.4.15)
mongodb-arb1 (mongodb 3.4.18)
mongodb-arb2 (mongodb 3.4.18)
mongodb-arb3 (mongodb 3.4.18)

I need to shutdown mongodb2 instance, so I'm trying to make one of the other DATA members become PRIMARY. I'm trying to stepDown , or shutdown mongodb2 , but nothing works ...

Below is the output of rs.stepDown() .

{
        "ok" : 0,
        "errmsg" : "No electable secondaries caught up as of 2018-12-07T16:36:04.546+0300. Please use {force: true} to force node to step down.",
        "code" : 50,
        "codeName" : "ExceededTimeLimit"
}

And when I use force:true , as db.adminCommand( { replSetStepDown: 120, secondaryCatchUpPeriodSecs:15, force: true } ) the output:

QUERY    [thread1] Error: error doing query: failed: network error while attempting to run command 'replSetStepDown' on host '127.0.0.1:27017'  :
DB.prototype.runCommand@src/mongo/shell/db.js:132:1
DB.prototype.adminCommand@src/mongo/shell/db.js:150:16
@(shell):1:1
2018-12-07T16:37:29.322+0300 I NETWORK  [thread1] trying reconnect to 127.0.0.1:27017 (127.0.0.1) failed
2018-12-07T16:37:29.324+0300 I NETWORK  [thread1] reconnect 127.0.0.1:27017 (127.0.0.1) ok

and the mongodb2 becomes SECONDARY , but becomes PRIMARY after a few seconds .

I'll provide the configuration below when I just POWER OFF the Primary Node , mongodb2.

[
            {
                    "_id" : 1,
                    "name" : "mongodb2:27017",
                    "health" : 0,
                    "state" : 8,
                    "stateStr" : "(not reachable/healthy)",
                    "uptime" : 0,
                    "optime" : {
                            "ts" : Timestamp(0, 0),
                            "t" : NumberLong(-1)
                    },
                    "optimeDurable" : {
                            "ts" : Timestamp(0, 0),
                            "t" : NumberLong(-1)
                    },
                    "optimeDate" : ISODate("1970-01-01T00:00:00Z"),
                    "optimeDurableDate" : ISODate("1970-01-01T00:00:00Z"),
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.187Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:22.796Z"),
                    "pingMs" : NumberLong(0),
                    "lastHeartbeatMessage" : "Connection refused",
                    "configVersion" : -1
            },
            {
                    "_id" : 3,
                    "name" : "mongodb4:27017",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 11771707,
                    "optime" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDate" : ISODate("2018-12-07T13:25:17Z"),
                    "infoMessage" : "could not find member to sync from",
                    "configVersion" : 11,
                    "self" : true
            },
            {
                    "_id" : 4,
                    "name" : "mongodb5:27017",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 11771703,
                    "optime" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDurable" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDate" : ISODate("2018-12-07T13:25:17Z"),
                    "optimeDurableDate" : ISODate("2018-12-07T13:25:17Z"),
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.161Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:53.141Z"),
                    "pingMs" : NumberLong(1),
                    "configVersion" : 11
            },
            {
                    "_id" : 5,
                    "name" : "mongodb6:27017",
                    "health" : 1,
                    "state" : 2,
                    "stateStr" : "SECONDARY",
                    "uptime" : 4982,
                    "optime" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDurable" : {
                            "ts" : Timestamp(1544189117, 1),
                            "t" : NumberLong(23)
                    },
                    "optimeDate" : ISODate("2018-12-07T13:25:17Z"),
                    "optimeDurableDate" : ISODate("2018-12-07T13:25:17Z"),
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:53.139Z"),
                    "pingMs" : NumberLong(0),
                    "configVersion" : 11
            },
            {
                    "_id" : 6,
                    "name" : "mongodb-arb:27017",
                    "health" : 1,
                    "state" : 7,
                    "stateStr" : "ARBITER",
                    "uptime" : 5462,
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:52.730Z"),
                    "pingMs" : NumberLong(1),
                    "configVersion" : 11
            },
            {
                    "_id" : 7,
                    "name" : "mongodb-arb:27018",
                    "health" : 1,
                    "state" : 7,
                    "stateStr" : "ARBITER",
                    "uptime" : 295,
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:52.726Z"),
                    "pingMs" : NumberLong(0),
                    "configVersion" : 11
            },
            {
                    "_id" : 8,
                    "name" : "mongodb-arb:27019",
                    "health" : 1,
                    "state" : 7,
                    "stateStr" : "ARBITER",
                    "uptime" : 162,
                    "lastHeartbeat" : ISODate("2018-12-07T13:25:53.162Z"),
                    "lastHeartbeatRecv" : ISODate("2018-12-07T13:25:52.773Z"),
                    "pingMs" : NumberLong(1),
                    "configVersion" : 11
            }
    ]

In this state , nothing changes , no matter how long i wait. The healthy DATA members just keep spitting the log :

2018-12-07T13:25:48.166+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Connecting to mongodb2:27017
2018-12-07T13:25:48.167+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Failed to connect to mongodb2:27017 - HostUnreachable: Connection refused
2018-12-07T13:25:48.167+0000 I ASIO     [NetworkInterfaceASIO-Replication-0] Dropping all pooled connections to mongodb2:27017 due to failed operation on a connection
2018-12-07T13:25:48.167+0000 I REPL     [ReplicationExecutor] Error in heartbeat request to mongodb2:27017; HostUnreachable: Connection refused

and there is no election of a PRIMARY node

  • What is your design goal with 3 arbiters? In general you should have either 0 or 1 arbiter. With 5 data bearing nodes you do not need an arbiter -- adding one actually adds potential failure scenarios. Since arbiters don't write data (but do count towards a voting majority) you have to consider the impact on write concerns. In the event one or more of your data bearing nodes are unavailable, there may be enough voting nodes to maintain a primary but insufficient nodes to acknowledge majority write concerns. This can lead to application stalls (waiting on write concern ack) and cache pressure. – Stennie Dec 7 '18 at 21:56
  • Also, you mention 8 instances but it looks like there are only 7 in your config and member list (4 data bearing + 3 arbiters). Ideally all should be running the same minor version of MongoDB server (eg. 3.4.18) rather than having a mix of versions. – Stennie Dec 7 '18 at 22:01
  • @Stennie hello , i was desperately trying to increase the number of healthy members and healthy member ratio , that's why i kept adding 2 more arbiters – kommradHomer Dec 7 '18 at 22:52
1

Turns out that , my 3 other DATA nodes all had "0" votes , and zero priority. Making them Non-Voting Members . Hence , when mongodb2 is down , there's not enough Voting Members to pick a primary node .

All aside , i don't know why my other Data nodes were given 0 priority and 0 votes , because i did nothing explicit to set them so.

  • There definitely is something very strange going on with your cluster. With 5 data bearing nodes (albeit I just see four), there simply is no need for an arbiter. Aside from that, having 5 data bearing nodes itself seems to be a bit odd to me. I could imagine 2 DC with two data bearing nodes each and an additional DC with an arbiter. In a single DC, the costs of having 5 nodes consistently running most likely exceeds the costs of a decent service level for replacement and 3 running nodes, TCOS taken into account. You only get an additional node which can fail per 2 data bearing nodes you add. – Markus W Mahlberg Dec 17 '18 at 16:16
  • @MarkusWMahlberg what do you mean with DC ? data center ? I had 5 data bearing nodes because 2 of them were on ARM scaleway instances , and 3 of them were on Virtual instances. we had 3 ARM data bearing instances in the beginning but they were problematic . Then we added 3 more data bearing nodes on scaleway virtual instances ,to see if we are doing something wrong and it has nothing to do with ARM instances maybe. But it was the ARM instances and the virtual instances had no problem . So i was trying to shutdown and remove the data bearing nodes on ARM instances.Hence the story begin – kommradHomer Dec 19 '18 at 15:39

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.