0

I've created a new 3 node replica set and rsync'ed data from an existing mongo database for initial population. The secondary nodes went into STARTUP2 mode and seemed to successfully sync from the primary. After poking around I've noticed that one of my collections has a data discrepancy between the PRIMARY and the 2 data baring SECONDARY nodes.

The rsync steps I followed were taken from this answer: https://stackoverflow.com/a/20937284/5274671

Some other rsync details are:

  • I sourced my data from an actively running mongodb database that was accepting reads and writes.
    • Data consistency between the source database and my replica set is not a priority. This replica set was stood up as a test to see how the replica set members behave and ensure replica set members all have consistent data replication.
  • The replica set member who the rsync was performed on was my primary member who is named ip-172-31-11-73:27017

Mongo Version: 3.0.15

Replica Set Status

bobby:PRIMARY> rs.status()
{
    "set" : "bobby",
    "date" : ISODate("2019-05-24T19:29:43.596Z"),
    "myState" : 1,
    "members" : [
        {
            "_id" : 0,
            "name" : "ip-172-31-11-73:27017",
            "health" : 1,
            "state" : 1,
            "stateStr" : "PRIMARY",
            "uptime" : 8767,
            "optime" : Timestamp(1558726180, 201),
            "optimeDate" : ISODate("2019-05-24T19:29:40Z"),
            "electionTime" : Timestamp(1558717420, 1),
            "electionDate" : ISODate("2019-05-24T17:03:40Z"),
            "configVersion" : 4,
            "self" : true
        },
        {
            "_id" : 1,
            "name" : "172.31.32.21:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 8763,
            "optime" : Timestamp(1558726180, 201),
            "optimeDate" : ISODate("2019-05-24T19:29:40Z"),
            "lastHeartbeat" : ISODate("2019-05-24T19:29:41.774Z"),
            "lastHeartbeatRecv" : ISODate("2019-05-24T19:29:42.237Z"),
            "pingMs" : 0,
            "syncingTo" : "172.31.42.124:27017",
            "configVersion" : 4
        },
        {
            "_id" : 2,
            "name" : "172.31.42.124:27017",
            "health" : 1,
            "state" : 2,
            "stateStr" : "SECONDARY",
            "uptime" : 8763,
            "optime" : Timestamp(1558726180, 201),
            "optimeDate" : ISODate("2019-05-24T19:29:40Z"),
            "lastHeartbeat" : ISODate("2019-05-24T19:29:43.202Z"),
            "lastHeartbeatRecv" : ISODate("2019-05-24T19:29:43.464Z"),
            "pingMs" : 1,
            "syncingTo" : "ip-172-31-11-73:27017",
            "configVersion" : 4
        }
    ],
    "ok" : 1
}

Replica Set Config

bobby:PRIMARY> rs.conf()
{
    "_id" : "bobby",
    "version" : 4,
    "members" : [
        {
            "_id" : 0,
            "host" : "ip-172-31-11-73:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        },
        {
            "_id" : 1,
            "host" : "172.31.32.21:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        },
        {
            "_id" : 2,
            "host" : "172.31.42.124:27017",
            "arbiterOnly" : false,
            "buildIndexes" : true,
            "hidden" : false,
            "priority" : 1,
            "tags" : {

            },
            "slaveDelay" : 0,
            "votes" : 1
        }
    ],
    "settings" : {
        "chainingAllowed" : true,
        "heartbeatTimeoutSecs" : 10,
        "getLastErrorModes" : {

        },
        "getLastErrorDefaults" : {
            "w" : "majority",
            "wtimeout" : 5500
        }
    }
}

Replica Set Slave Replication Info

bobby:PRIMARY> rs.printSlaveReplicationInfo()
source: 172.31.32.21:27017
    syncedTo: Fri May 24 2019 19:30:14 GMT+0000 (UTC)
    0 secs (0 hrs) behind the primary 
source: 172.31.42.124:27017
    syncedTo: Fri May 24 2019 19:30:14 GMT+0000 (UTC)
    0 secs (0 hrs) behind the primary

When running db.collection.count() the primary returns 4463665 and both secondary nodes return 4463658.

  • Is this normal?
  • What is the root cause of the data discrepancy?
  • Is there a way to sync up this collection so that the secondary nodes contain all primary data?

Notes:

  • My replia set is not being actively written to or read from at the moment of this discovery.
  • I've read and applied this question: different db stats in MongoDB replica set using db.collection.find({_id:{ $gt: MinKey, $lt:MaxKey} }, {_id:1}).itcount() resulted in the same data discrepancy.
  • Can you edit your question to describe this process: rsync'ed data from an existing mongo database for initial population. For example, which replica set members did you rsync data from/to? Did you stop writes on the source replica set? Also note that MongoDB 3.0.15 was released in May, 2017 and the 3.0 release series reached end of life in February 2018. I expect you are using the same version as your source replica set, but you should definitely plan to upgrade to an actively supported version of MongoDB (currently 3.4 or newer). – Stennie May 25 at 4:16
  • @Stennie I've added more rsync details. Also I would love to upgrade our MongoDB to a newer version but unfortunately it is not an option for me at the moment. Do you think that maybe since I rsync'ed from an actively running Mongo instance that some of the copied data was not fully committed to mongo and therefore was corrupt and then not replicated to the secondary members? – Chris May 28 at 15:23
  • Unless you stop writes using either db.fsyncLock() or by shutting down the mongod you are copying files from, rsync isn't a recommendable backup solution as you are likely to get an inconsistent file copy. The MongoDB documentation includes information on supported Backup Methods and how to Resync a member of a replica set. – Stennie May 29 at 0:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.