0

I have a MongoDB (3.2.6) replica set with two nodes and an arbiter. The secondary node was out for quite a long time and was recently brought back up with an initial sync. So at the moment the rs.status() is as follows:

rs.status()

{
    "set" : "anonymizedReplicaSet",
    "date" : ISODate("2017-05-24T07:33:08.629Z"),
    "myState" : 2,
    "term" : NumberLong(-1),
    "syncingTo" : "anonymized01:27017",
    "heartbeatIntervalMillis" : NumberLong(2000),
    "members" : [ 
        {
            "_id" : 1,
            "name" : "anonymized01:27017",
            "health" : 1.0,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 157061,
            "optime" : Timestamp(6423603132856545, 12785),
            "optimeDate" : ISODate("2017-05-24T07:40:52.000Z"),
            "lastHeartbeat" : ISODate("2017-05-24T07:33:06.790Z"),
            "lastHeartbeatRecv" : ISODate("2017-05-24T07:33:06.790Z"),
            "pingMs" : NumberLong(0),
            "electionTime" : Timestamp(6415474245823889, 1),
            "electionDate" : ISODate("2017-05-02T09:56:38.000Z"),
            "configVersion" : 10
        }, 
        {
            "_id" : 2,
            "name" : "anonymized02:27017",
            "health" : 1.0,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 157062,
            "optime" : Timestamp(6423603141446477, 10075),
            "optimeDate" : ISODate("2017-05-24T07:40:54.000Z"),
            "syncingTo" : "anonymized01:27017",
            "configVersion" : 10,
            "self" : true
        }, 
        {
            "_id" : 3,
            "name" : "anonymizedas:27017",
            "health" : 1.0,
            "state" : 7,
            "stateStr" : "ARBITER",
            "uptime" : 157061,
            "lastHeartbeat" : ISODate("2017-05-24T07:33:06.790Z"),
            "lastHeartbeatRecv" : ISODate("2017-05-24T07:33:07.248Z"),
            "pingMs" : NumberLong(0),
            "configVersion" : 10
        }
    ],
    "ok" : 1.0
}

However, I noticed that the two nodes do not agree on their stats, e.g., when I issue db.stats() simultaneously to the two servers I get:

Primary Node db.stats()

{
    "db" : "mydb",
    "collections" : 8,
    "objects" : 43933967,
    "avgObjSize" : 575.327071124718,
    "dataSize" : 25276400557.0,
    "storageSize" : 11909304320.0,
    "numExtents" : 0,
    "indexes" : 12,
    "indexSize" : 997511168.0,
    "ok" : 1.0
}

Secondary Node db.stats()

{
    "db" : "mydb",
    "collections" : 8,
    "objects" : 44016455,
    "avgObjSize" : 576.927973845236,
    "dataSize" : 25394324199.0,
    "storageSize" : 9882480640.0,
    "numExtents" : 0,
    "indexes" : 12,
    "indexSize" : 901128192.0,
    "ok" : 1.0
}

Shouldn't the number of objects on the two nodes agree? In addition, I receive different results when issuing mydb.mycollection.count() commands. Is this normal behavior? Or is something wrong with the syncing of the two nodes?

  • ,How yor secondary replica configure with primary? "Secondary members replicate data continuously after the initial sync." or Secondary members copy the oplog from their sync from source and apply these operations in an asynchronous process. – Md Haidar Ali Khan May 24 '17 at 11:56
  • @MdHaidarAliKhan I believe that replication is synchronous, issuing rs.printSlaveReplicationInfo() from the primary says that secondary is 0 secs (0 hrs) behind the primary to 2 secs (0 hrs) behind the primary – Panagiotis May 24 '17 at 12:21
  • Did you check through "connectionStatus" administrative commands. what is showing current status of your replioca? – Md Haidar Ali Khan May 24 '17 at 13:12
  • @MdHaidarAliKhan I get "ok" : 1.0 – Panagiotis May 24 '17 at 13:17
  • That means at current status of both replica in 'synchronization' state. – Md Haidar Ali Khan May 24 '17 at 13:19
0

Logically answer is YES.

When I tested on our multi terabyte DB's, I got differences between replica set members, however this was measuring mistake, error on the procedure. f.ex. db.stats() is not good command, because RS node can have db.system.profile -collection, what is NOT replicated to other nodes.

How to do it right (do these on all replica set nodes):

  1. Stop all write traffic to MongoDB, you can do that by closing all clients OR giving (at Primary) command db.fsyncLock()
  2. select right DB (use xyz) and give command db.setSlaveOk(true)
  3. Don't use db.collection.find().count() or db.collection.count(), use command db.collection.find({_id:{ $gt: MinKey, $lt:MaxKey} }, {_id:1}).itcount() to count how many documents there is
  4. After you have done these itcount() commands on multiple collections and you are happy (or not) to result.. Don't forget to give (at primary) command db.fsyncUnlock()

we had maintenance break and I did this counting procedure during it and got (this time) right result. Every RS node had same count of documents. ;-)

  • Thanks for the feedback, please let me know if you come up with further info on the matter. – Panagiotis May 25 '17 at 7:10
  • Hi! I edit my answer... – JJussi May 30 '17 at 2:09
0

I would not discard anything of @JJussi, May be the latest oplog timestamp on the secondary node is smaller than the oldest oplog timestamp on the primary node.

Recently, there have been many cases of too-high QPS on the primary node in the production environment, making the secondary node unable to keep up the synchronization with the primary node (the latest oplog timestamp on the secondary node is smaller than the oldest oplog timestamp on the primary node), making the secondary node change to the RECOVERING status. This problem requiring human intervention, namely sending the resync command to the secondary node to enforce the secondary node to perform a full re-synchronization.

the secondary node will perform an initial sync operation first, that is the full data synchronization to traverse all the sets of all the databases on the primary node and copy the data to its own node. Then it reads the oplog dated “within the period from full synchronization beginning to end” and replay the oplog.

For your further ref Here and Here

  • thanks for your answer, the status of my secondary node is SECONDARY as shown above, not RECOVERING, also the objects on the db.stats() results and the counts I have issued appear to be larger on the secondary node, so I don't think that the secondary node had trouble catching up with the primary – Panagiotis May 24 '17 at 13:36
  • @Panagiotis,Yes. Because through the administrative command "connectionStatus" we have checked . As per it's showing the status 1. I have just explained the synchronization process. How it happend in MongoDB. – Md Haidar Ali Khan May 24 '17 at 13:38
  • Not sure I can follow you, that is not how it happened in my case, I performed an initial sync, the node reached SECONDARY state and has not fallen to another state since, I am just puzzled by the different db.stats() that I receive, especially since the secondary node appears to have more objects. – Panagiotis May 24 '17 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.