0

I have 3 mongodb servers in our production environment with replica set configured. All the 3 are running in 3 different Ec2 instances. The application server is running in another instance. The server connects to primary monogdb for write. And the read preference is set to Primary so it should contact primary for read too.

Everything was working normal until one day suddenly the write operations on primary stopped working. I was not able to update, insert anything through command line. Even our application stopped working (user were able to view the page but not do any actions).

The application was down for 15 minutes and i had to restart the mongo instance for it to resume working properly. Any idea why this happens and how to prevent this from happening in the future?

  • Did you checked.What is showing the status of replica set through "rs.status()" of primary server. In that node what is the output from rs.isMaster(). Are you sure you have connected with Primary replica or else. – Md Haidar Ali Khan Apr 12 '18 at 7:03
  • Hi ajit, could you please add MongoDB version – Priyanka Kariya Feb 11 at 10:55
0

Why does mongodb primary stops writing sometimes?

As per MongoDB BOL here In the following 3-member replica set, the primary becomes unavailable. In some circumstances, two nodes in a replica set may transiently believe that they are the primary, but at most, one of them will be able to complete writes with { w: "majority" } write concern. The node that can complete { w: "majority" } writes is the current primary, and the other node is a former primary that has not yet recognized its demotion, typically due to a network partition. When this occurs, clients that connect to the former primary may observe stale data despite having requested read preference primary, and new writes to the former primary will eventually roll back.

As the primary is the only member in the replica set that receives write operations. MongoDB applies write operations on the primary and then records the operations on the primary’s oplog. Secondary members replicate this log and apply the operations to their data sets.

All members of the replica set can accept read operations. However, by default, an application directs its read operations to the primary member.

You can check the status of all replica set through isMaster, rs.status() and replSetGetStatus through db.adminCommand( { replSetGetStatus : 1 } ).

For example here, I am write down the command to check the health status report for replica set status through rs.isMaster() , rs.status() as well as db.adminCommand( { replSetGetStatus : 1 } ) command.

Replica set reports through rs.isMaster() command

MongoDB Enterprise repltest:PRIMARY> rs.isMaster()
{
        "hosts" : [
                "192.168.103.XXX:26XX1",
                "192.168.103.XXX:26XX2",
                "192.168.103.XXX:26XX3"
        ],
        "setName" : "repltest",
        "setVersion" : 3,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "192.168.103.XXX:26XX1",
        "me" : "192.168.103.XXX:26XX1",
        "electionId" : ObjectId("7fffffff0000000000000010"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1523517219, 1),
                        "t" : NumberLong(16)
                },
                "lastWriteDate" : ISODate("2018-04-12T07:13:39Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1523517219, 1),
                        "t" : NumberLong(16)
                },
                "majorityWriteDate" : ISODate("2018-04-12T07:13:39Z")
        },
        "configsvr" : 2,
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2018-04-12T07:13:44.956Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "minWireVersion" : 0,
        "maxWireVersion" : 6,
        "readOnly" : false,
        "ok" : 1,
        "operationTime" : Timestamp(1523517219, 1),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000010")
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1523517219, 1),
                "signature" : {
                        "hash" : BinData(0,"1pRzHFFbcmxMaD+2KxZ5XjidqNc="),
                        "keyId" : NumberLong("6537909663055216660")
                }
        }
}

Replica set reports through rs.status() command

MongoDB Enterprise repltest:PRIMARY> rs.status()
{
        "set" : "repltest",
        "date" : ISODate("2018-04-12T07:13:25.891Z"),
        "myState" : 1,
        "term" : NumberLong(16),
        "configsvr" : true,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.103.XXX:26XX1",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 85,
                        "optime" : {
                                "ts" : Timestamp(1523517205, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:13:25Z"),
                        "electionTime" : Timestamp(1523517143, 1),
                        "electionDate" : ISODate("2018-04-12T07:12:23Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "192.168.103.XXX:26XX2",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 69,
                        "optime" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:13:19Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:13:19Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:13:25.203Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:13:25.556Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX1",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "192.168.103.XXX:26XX3",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 58,
                        "optime" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:13:19Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:13:19Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:13:25.260Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:13:24.165Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX2",
                        "configVersion" : 3
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1523517205, 1),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000010")
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1523517205, 1),
                "signature" : {
                        "hash" : BinData(0,"oWtGCmbOFVAtaIEfncEX7YVU680="),
                        "keyId" : NumberLong("6537909663055216660")
                }
        }
}

Replica set reports through db.adminCommand( { replSetGetStatus : 1 } ) command

MongoDB Enterprise repltest:PRIMARY> db.adminCommand( { replSetGetStatus : 1 } )
{
        "set" : "repltest",
        "date" : ISODate("2018-04-12T07:14:50.139Z"),
        "myState" : 1,
        "term" : NumberLong(16),
        "configsvr" : true,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.103.XXX:26XX1",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 170,
                        "optime" : {
                                "ts" : Timestamp(1523517289, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:14:49Z"),
                        "electionTime" : Timestamp(1523517143, 1),
                        "electionDate" : ISODate("2018-04-12T07:12:23Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "192.168.103.XXX:26XX2",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 153,
                        "optime" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:14:39Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:14:39Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:14:49.294Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:14:49.680Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX1",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "192.168.103.XXX:26XX3",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 142,
                        "optime" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:14:39Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:14:39Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:14:49.355Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:14:48.240Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX2",
                        "configVersion" : 3
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1523517289, 1),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000010")
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1523517289, 1),
                "signature" : {
                        "hash" : BinData(0,"oOpHG6ldi67Zm2pc6Rfd0BhRsCU="),
                        "keyId" : NumberLong("6537909663055216660")
                }
        }
}

Note: All command here I have ran against the Primary Replica Node. This replica set reports for test environment. Here i have change some IP & Port Number with XXX , due to security reason.

For your further ref here

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.