Why does mongodb primary stops writing sometimes?

Question

I have 3 mongodb servers in our production environment with replica set configured. All the 3 are running in 3 different Ec2 instances. The application server is running in another instance. The server connects to primary monogdb for write. And the read preference is set to Primary so it should contact primary for read too.

Everything was working normal until one day suddenly the write operations on primary stopped working. I was not able to update, insert anything through command line. Even our application stopped working (user were able to view the page but not do any actions).

The application was down for 15 minutes and i had to restart the mongo instance for it to resume working properly. Any idea why this happens and how to prevent this from happening in the future?

Answer 1

Why does mongodb primary stops writing sometimes?

As per MongoDB BOL here In the following 3-member replica set, the primary becomes unavailable. In some circumstances, two nodes in a replica set may transiently believe that they are the primary, but at most, one of them will be able to complete writes with { w: "majority" } write concern. The node that can complete { w: "majority" } writes is the current primary, and the other node is a former primary that has not yet recognized its demotion, typically due to a network partition. When this occurs, clients that connect to the former primary may observe stale data despite having requested read preference primary, and new writes to the former primary will eventually roll back.

As the primary is the only member in the replica set that receives write operations. MongoDB applies write operations on the primary and then records the operations on the primary’s oplog. Secondary members replicate this log and apply the operations to their data sets.

All members of the replica set can accept read operations. However, by default, an application directs its read operations to the primary member.

You can check the status of all replica set through isMaster, rs.status() and replSetGetStatus through db.adminCommand( { replSetGetStatus : 1 } ).

For example here, I am write down the command to check the health status report for replica set status through rs.isMaster() , rs.status() as well as db.adminCommand( { replSetGetStatus : 1 } ) command.

Replica set reports through rs.isMaster() command

MongoDB Enterprise repltest:PRIMARY> rs.isMaster()
{
        "hosts" : [
                "192.168.103.XXX:26XX1",
                "192.168.103.XXX:26XX2",
                "192.168.103.XXX:26XX3"
        ],
        "setName" : "repltest",
        "setVersion" : 3,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "192.168.103.XXX:26XX1",
        "me" : "192.168.103.XXX:26XX1",
        "electionId" : ObjectId("7fffffff0000000000000010"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1523517219, 1),
                        "t" : NumberLong(16)
                },
                "lastWriteDate" : ISODate("2018-04-12T07:13:39Z"),
                "majorityOpTime" : {
                        "ts" : Timestamp(1523517219, 1),
                        "t" : NumberLong(16)
                },
                "majorityWriteDate" : ISODate("2018-04-12T07:13:39Z")
        },
        "configsvr" : 2,
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 100000,
        "localTime" : ISODate("2018-04-12T07:13:44.956Z"),
        "logicalSessionTimeoutMinutes" : 30,
        "minWireVersion" : 0,
        "maxWireVersion" : 6,
        "readOnly" : false,
        "ok" : 1,
        "operationTime" : Timestamp(1523517219, 1),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000010")
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1523517219, 1),
                "signature" : {
                        "hash" : BinData(0,"1pRzHFFbcmxMaD+2KxZ5XjidqNc="),
                        "keyId" : NumberLong("6537909663055216660")
                }
        }
}

Replica set reports through rs.status() command

MongoDB Enterprise repltest:PRIMARY> rs.status()
{
        "set" : "repltest",
        "date" : ISODate("2018-04-12T07:13:25.891Z"),
        "myState" : 1,
        "term" : NumberLong(16),
        "configsvr" : true,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1523517205, 1),
                        "t" : NumberLong(16)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.103.XXX:26XX1",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 85,
                        "optime" : {
                                "ts" : Timestamp(1523517205, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:13:25Z"),
                        "electionTime" : Timestamp(1523517143, 1),
                        "electionDate" : ISODate("2018-04-12T07:12:23Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "192.168.103.XXX:26XX2",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 69,
                        "optime" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:13:19Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:13:19Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:13:25.203Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:13:25.556Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX1",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "192.168.103.XXX:26XX3",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 58,
                        "optime" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517199, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:13:19Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:13:19Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:13:25.260Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:13:24.165Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX2",
                        "configVersion" : 3
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1523517205, 1),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000010")
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1523517205, 1),
                "signature" : {
                        "hash" : BinData(0,"oWtGCmbOFVAtaIEfncEX7YVU680="),
                        "keyId" : NumberLong("6537909663055216660")
                }
        }
}

Replica set reports through db.adminCommand( { replSetGetStatus : 1 } ) command

MongoDB Enterprise repltest:PRIMARY> db.adminCommand( { replSetGetStatus : 1 } )
{
        "set" : "repltest",
        "date" : ISODate("2018-04-12T07:14:50.139Z"),
        "myState" : 1,
        "term" : NumberLong(16),
        "configsvr" : true,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                },
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                },
                "appliedOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1523517289, 1),
                        "t" : NumberLong(16)
                }
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "192.168.103.XXX:26XX1",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 170,
                        "optime" : {
                                "ts" : Timestamp(1523517289, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:14:49Z"),
                        "electionTime" : Timestamp(1523517143, 1),
                        "electionDate" : ISODate("2018-04-12T07:12:23Z"),
                        "configVersion" : 3,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "192.168.103.XXX:26XX2",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 153,
                        "optime" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:14:39Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:14:39Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:14:49.294Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:14:49.680Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX1",
                        "configVersion" : 3
                },
                {
                        "_id" : 2,
                        "name" : "192.168.103.XXX:26XX3",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 142,
                        "optime" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1523517279, 1),
                                "t" : NumberLong(16)
                        },
                        "optimeDate" : ISODate("2018-04-12T07:14:39Z"),
                        "optimeDurableDate" : ISODate("2018-04-12T07:14:39Z"),
                        "lastHeartbeat" : ISODate("2018-04-12T07:14:49.355Z"),
                        "lastHeartbeatRecv" : ISODate("2018-04-12T07:14:48.240Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "192.168.103.XXX:26XX2",
                        "configVersion" : 3
                }
        ],
        "ok" : 1,
        "operationTime" : Timestamp(1523517289, 1),
        "$gleStats" : {
                "lastOpTime" : Timestamp(0, 0),
                "electionId" : ObjectId("7fffffff0000000000000010")
        },
        "$clusterTime" : {
                "clusterTime" : Timestamp(1523517289, 1),
                "signature" : {
                        "hash" : BinData(0,"oOpHG6ldi67Zm2pc6Rfd0BhRsCU="),
                        "keyId" : NumberLong("6537909663055216660")
                }
        }
}

Note: All command here I have ran against the Primary Replica Node. This replica set reports for test environment. Here i have change some IP & Port Number with XXX , due to security reason.

For your further ref here