0

We have 5node replicaset , in which 2 servers are in Jacksonville datacenter and 2 servers are on Atlanta datacenter and an arbiter . I had brought down the Atlanta servers , but the write operations are getting in queue as huge ., but if we bring back any of the Atlanta servers, the queue is getting settle down . The writeConcern is set to 1, still the issue persist , Please suggest on this issue .

PRIMARY> rs.status()
{
        "set" : "MongoUAT_rs",
        "date" : ISODate("2021-10-19T12:18:11.103Z"),
        "myState" : 1,
        "term" : NumberLong(52),
        "syncSourceHost" : "",
        "syncSourceId" : -1,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "majorityVoteCount" : 3,
        "writeMajorityCount" : 3,
        "votingMembersCount" : 5,
        "writableVotingMembersCount" : 4,
        "optimes" : {
                "lastCommittedOpTime" : {
                        "ts" : Timestamp(1634645886, 2),
                        "t" : NumberLong(52)
                },
                "lastCommittedWallTime" : ISODate("2021-10-19T12:18:06.088Z"),
                "readConcernMajorityOpTime" : {
                        "ts" : Timestamp(1634645886, 2),
                        "t" : NumberLong(52)
                },
                "readConcernMajorityWallTime" : ISODate("2021-10-19T12:18:06.088Z"),
                "appliedOpTime" : {
                        "ts" : Timestamp(1634645886, 2),
                        "t" : NumberLong(52)
                },
                "durableOpTime" : {
                        "ts" : Timestamp(1634645886, 2),
                        "t" : NumberLong(52)
                },
                "lastAppliedWallTime" : ISODate("2021-10-19T12:18:06.088Z"),
                "lastDurableWallTime" : ISODate("2021-10-19T12:18:06.088Z")
        },
        "lastStableRecoveryTimestamp" : Timestamp(1634645881, 1),
        "electionCandidateMetrics" : {
                "lastElectionReason" : "electionTimeout",
                "lastElectionDate" : ISODate("2021-09-15T07:19:12.893Z"),
                "electionTerm" : NumberLong(52),
                "lastCommittedOpTimeAtElection" : {
                        "ts" : Timestamp(1631690346, 1),
                        "t" : NumberLong(51)
                },
                "lastSeenOpTimeAtElection" : {
                        "ts" : Timestamp(1631690346, 1),
                        "t" : NumberLong(51)
                },
                "numVotesNeeded" : 3,
                "priorityAtElection" : 2,
                "electionTimeoutMillis" : NumberLong(10000),
                "priorPrimaryMemberId" : 2,
                "numCatchUpOps" : NumberLong(0),
                "newTermStartDate" : ISODate("2021-09-15T07:19:13.204Z"),
                "wMajorityWriteAvailabilityDate" : ISODate("2021-09-15T07:19:14.073Z")
        },
        "electionParticipantMetrics" : {
                "votedForCandidate" : true,
                "electionTerm" : NumberLong(51),
                "lastVoteDate" : ISODate("2021-09-15T06:56:52.839Z"),
                "electionCandidateMemberId" : 2,
                "voteReason" : "",
                "lastAppliedOpTimeAtElection" : {
                        "ts" : Timestamp(1631689053, 1189),
                        "t" : NumberLong(50)
                },
                "maxAppliedOpTimeInSet" : {
                        "ts" : Timestamp(1631689053, 1189),
                        "t" : NumberLong(50)
                },
                "priorityAtElection" : 2
        },
        "members" : [
                {
                        "_id" : 0,
                        "name" : "atlmongouat2:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 406229,
                        "optime" : {
                                "ts" : Timestamp(1634645423, 1125),
                                "t" : NumberLong(52)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1634645421, 1919),
                                "t" : NumberLong(52)
                        },
                        "optimeDate" : ISODate("2021-10-19T12:10:23Z"),
                        "optimeDurableDate" : ISODate("2021-10-19T12:10:21Z"),
                        "lastHeartbeat" : ISODate("2021-10-19T12:18:10.794Z"),
                        "lastHeartbeatRecv" : ISODate("2021-10-19T12:18:10.857Z"),
                        "pingMs" : NumberLong(7),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "atlmongouat1:27018",
                        "syncSourceId" : 1,
                        "infoMessage" : "",
                        "configVersion" : 18,
                        "configTerm" : 52
                },
                {
                        "_id" : 1,
                        "name" : "atlmongouat1:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 2421252,
                        "optime" : {
                                "ts" : Timestamp(1634645886, 2),
                                "t" : NumberLong(52)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1634645886, 2),
                                "t" : NumberLong(52)
                        },
                        "optimeDate" : ISODate("2021-10-19T12:18:06Z"),
                        "optimeDurableDate" : ISODate("2021-10-19T12:18:06Z"),
                        "lastHeartbeat" : ISODate("2021-10-19T12:18:09.717Z"),
                        "lastHeartbeatRecv" : ISODate("2021-10-19T12:18:10.660Z"),
                        "pingMs" : NumberLong(7),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "jaxmongouat2:27018",
                        "syncSourceId" : 3,
                        "infoMessage" : "",
                        "configVersion" : 18,
                        "configTerm" : 52
                },
                {
                        "_id" : 2,
                        "name" : "jaxmongouat1:27018",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 4431183,
                        "optime" : {
                                "ts" : Timestamp(1634645886, 2),
                                "t" : NumberLong(52)
                        },
                        "optimeDurable" : {
                                "ts" : Timestamp(1634645886, 2),
                                "t" : NumberLong(52)
                        },
                        "optimeDate" : ISODate("2021-10-19T12:18:06Z"),
                        "optimeDurableDate" : ISODate("2021-10-19T12:18:06Z"),
                        "lastHeartbeat" : ISODate("2021-10-19T12:18:09.568Z"),
                        "lastHeartbeatRecv" : ISODate("2021-10-19T12:18:10.720Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "jaxmongouat2:27018",
                        "syncSourceId" : 3,
                        "infoMessage" : "",
                        "configVersion" : 18,
                        "configTerm" : 52
                },
                {
                        "_id" : 3,
                        "name" : "jaxmongouat2:27018",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 4431190,
                        "optime" : {
                                "ts" : Timestamp(1634645886, 2),
                                "t" : NumberLong(52)
                        },
                        "optimeDate" : ISODate("2021-10-19T12:18:06Z"),
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "electionTime" : Timestamp(1631690352, 1),
                        "electionDate" : ISODate("2021-09-15T07:19:12Z"),
                        "configVersion" : 18,
                        "configTerm" : 52,
                        "self" : true,
                        "lastHeartbeatMessage" : ""
                },
                {
                        "_id" : 4,
                        "name" : "mongoarbuat1:27018",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 718877,
                        "lastHeartbeat" : ISODate("2021-10-19T12:18:10.681Z"),
                        "lastHeartbeatRecv" : ISODate("2021-10-19T12:18:10.453Z"),
                        "pingMs" : NumberLong(0),
                        "lastHeartbeatMessage" : "",
                        "syncSourceHost" : "",
                        "syncSourceId" : -1,
                        "infoMessage" : "",
                        "configVersion" : 18,
                        "configTerm" : 52
                }
        ],
        "ok" : 1,
        "$clusterTime" : {
                "clusterTime" : Timestamp(1634645886, 2),
                "signature" : {
                        "hash" : BinData(0,"1yROW24X4mU50Deh8QQFUjHZpKc="),
                        "keyId" : NumberLong("6963265726354292738")
                }
        },
        "operationTime" : Timestamp(1634645886, 2)
}

thanks

11
  • Which version of MongoDB?
    – Joe
    Oct 19 at 1:38
  • You may want to include the output of the rs.status() in your question post.
    – prasad_
    Oct 19 at 4:30
  • thanks Joe, for checking . mongodb version is 4.4.4 ( community version) Oct 19 at 12:29
  • What is the output of db.adminCommand({ replSetGetConfig: 1 }).config Oct 19 at 14:15
  • Is your arbiter located in Atlanta DC? You may deploy it in a third location. Oct 19 at 14:18
1

MongoDB 4.4 has read concern majority enabled by default. This means that each node will keep a snapshot of the most recent data confirmed to be committed to a majority of the replica set.

In a 5 node replica set, data must be written to 3 nodes to be considered majority committed.

When you have 2 nodes down, the cluster cannot commit anything to a majority of nodes, which leads to cache and memory pressure.

To prevent this, either convert the arbiter to a data-bearing node, or set enableMajorityReadConcern to false in the config file.

3
  • thanks Joe, it sounds good , Oct 22 at 19:00
  • anyway to check the enableMajorityReadConcern , if it is enabled or not . also, is it possible to change this parameter dynamically ? Oct 22 at 19:01
  • I think it can only be set at startup. Run db.adminCommand("getCmdLineOpts") and look in the 'parsed' section. If it is not listed, then it is enabled.
    – Joe
    Oct 22 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.