2

Node logs under Recovery mode, even after restarting:

    2018-04-24T12:37:52.915+0530 I REPL [replication-0] We are too stale to use J-DB-02:27017 as a sync source. Blacklisting this sync source because our last fetched timestamp: 5ade01e0:66 is before their earliest timestamp: 5aded094:198 for 1min until: 2018-04-24T12:38:52.915+0530 2018-04-24T12:37:52.916+0530 I REPL [replication-0] could not find member to sync from 2018-04-24T12:37:52.916+0530 E REPL [rsBackgroundSync] too stale to catch up -- entering maintenance mode 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] Our newest OpTime : { ts: Timestamp 1524498912000|102, t: 68 } 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] Earliest OpTime available is { ts: Timestamp 1524551828000|408, t: 68 } 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] See http://dochub.mongodb.org/core/resyncingaverystalereplicasetmember 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] going into maintenance mode with 0 other maintenance mode tasks in progress 2018-04-24T12:37:52.916+0530 I REPL [rsBackgroundSync] transition to RECOVERING

Here is the status of replica set from primary server

    WAPRS01:PRIMARY> db.oplog.rs.stats().maxSize
    WAPRS01:PRIMARY> rs.status();
    {
            "set" : "WAPRS01",
            "date" : ISODate("2018-04-24T07:17:15.276Z"),
            "myState" : 1,
            "term" : NumberLong(68),
            "heartbeatIntervalMillis" : NumberLong(2000),
            "optimes" : {
                    "lastCommittedOpTime" : {
                            "ts" : Timestamp(1524498912, 102),
                            "t" : NumberLong(68)
                    },
                    "appliedOpTime" : {
                            "ts" : Timestamp(1524554235, 160),
                            "t" : NumberLong(68)
                    },
                    "durableOpTime" : {
                            "ts" : Timestamp(1524554235, 92),
                            "t" : NumberLong(68)
                    }
            },
            "members" : [
                    {
                            "_id" : 0,
                            "name" : "J-DB-01:27017",
                            "health" : 1,
                            "state" : 3,
                            "stateStr" : "RECOVERING",
                            "uptime" : 569,
                            "optime" : {
                                    "ts" : Timestamp(1524498912, 102),
                                    "t" : NumberLong(68)
                            },
                            "optimeDurable" : {
                                    "ts" : Timestamp(1524498912, 102),
                                    "t" : NumberLong(68)
                            },
                            "optimeDate" : ISODate("2018-04-23T15:55:12Z"),
                            "optimeDurableDate" : ISODate("2018-04-23T15:55:12Z"),
                            "lastHeartbeat" : ISODate("2018-04-24T07:17:13.532Z"),
                            "lastHeartbeatRecv" : ISODate("2018-04-24T07:17:13.493Z"),
                            "pingMs" : NumberLong(0),
                            "configVersion" : 4
                    },
                    {
                            "_id" : 1,
                            "name" : "J-DB-02:27017",
                            "health" : 1,
                            "state" : 2,
                            "stateStr" : "SECONDARY",
                            "uptime" : 299382,
                            "optime" : {
                                    "ts" : Timestamp(1524554232, 270),
                                    "t" : NumberLong(68)
                            },
                            "optimeDurable" : {
                                    "ts" : Timestamp(1524554232, 270),
                                    "t" : NumberLong(68)
                            },
                            "optimeDate" : ISODate("2018-04-24T07:17:12Z"),
                            "optimeDurableDate" : ISODate("2018-04-24T07:17:12Z"),
                            "lastHeartbeat" : ISODate("2018-04-24T07:17:13.454Z"),
                            "lastHeartbeatRecv" : ISODate("2018-04-24T07:17:13.864Z"),
                            "pingMs" : NumberLong(0),
                            "syncingTo" : "J-DB-03:27017",
                            "configVersion" : 4
                    },
                    {
                            "_id" : 2,
                            "name" : "J-DB-03:27017",
                            "health" : 1,
                            "state" : 1,
                            "stateStr" : "PRIMARY",
                            "uptime" : 613420,
                            "optime" : {
                                    "ts" : Timestamp(1524554235, 160),
                                    "t" : NumberLong(68)
                            },
                            "optimeDate" : ISODate("2018-04-24T07:17:15Z"),
                            "electionTime" : Timestamp(1524254862, 1),
                            "electionDate" : ISODate("2018-04-20T20:07:42Z"),
                            "configVersion" : 4,
                            "self" : true
                    },
                    {
                            "_id" : 3,
                            "name" : "J-DB-04:27017",
                            "health" : 1,
                            "state" : 7,
                            "stateStr" : "ARBITER",
                            "uptime" : 300361,
                            "lastHeartbeat" : ISODate("2018-04-24T07:17:14.863Z"),
                            "lastHeartbeatRecv" : ISODate("2018-04-24T07:17:14.229Z"),
                            "pingMs" : NumberLong(0),
                            "configVersion" : 4
                    }
            ],`enter code here`
            "ok" : 1
    }
    WAPRS01:PRIMARY>

db.isMaster():

WAPRS01:PRIMARY> db.isMaster();
{
        "hosts" : [
                "J-DB-01:27017",
                "J-DB-02:27017",
                "J-DB-03:27017"
        ],
        "arbiters" : [
                "J-DB-04:27017"
        ],
        "setName" : "WAPRS01",
        "setVersion" : 4,
        "ismaster" : true,
        "secondary" : false,
        "primary" : "J-DB-03:27017",
        "me" : "J-DB-03:27017",
        "electionId" : ObjectId("7fffffff0000000000000044"),
        "lastWrite" : {
                "opTime" : {
                        "ts" : Timestamp(1524559624, 465),
                        "t" : NumberLong(68)
                },
                "lastWriteDate" : ISODate("2018-04-24T08:47:04Z")
        },
        "maxBsonObjectSize" : 16777216,
        "maxMessageSizeBytes" : 48000000,
        "maxWriteBatchSize" : 1000,
        "localTime" : ISODate("2018-04-24T08:47:04.511Z"),
        "maxWireVersion" : 5,
        "minWireVersion" : 0,
        "readOnly" : false,
        "ok" : 1
}
  • ,Are you sure . Is it Primary Replica Set reports status. Did you checked through "db.isMaster()".Through which you can confirm your Primary Replica as well as Secondary replica. – Md Haidar Ali Khan Apr 24 '18 at 8:46
  • have added db.isMaster() results also above – Rishi Tyagi Apr 24 '18 at 8:48
  • @ RishiTyagi,Yes, As per db.isMaster() reports "ismaster" : true. That Replica Set Server is Primary. – Md Haidar Ali Khan Apr 24 '18 at 8:50
  • you are right, I had added above status of primary server and error logs from the replica server which is causing issues – Rishi Tyagi Apr 24 '18 at 8:52
  • ,As from your rs.status() reports, i am able to see your one of secondary replica node is showing in "Recovering" state.please post mongodb.log from RECOVERING nodes. – Md Haidar Ali Khan Apr 24 '18 at 10:51
3

As I am able to see from your rs.status() , there is secondary replica is showing in RECOVERING state. And as MongoDB jira blog After node goes into RECOVERING due to being too stale to sync from its source, it will never recover even if there is a valid sync source it could use.

As per MongoDB documentation here A replica set member becomes “stale” when its replication process falls so far behind that the primary overwrites oplog entries the member has not yet replicated. The member cannot catch up and becomes “stale.” When this occurs, you must completely resynchronize the member by removing its data and performing an initial sync.

For further your ref here , here , here, here and here

  • This is the state again after reading 75% data post resynch is started by clearing all data on the server, Initially it was showing StartUP-2 state and then went into RECOVERING State again – Rishi Tyagi Apr 24 '18 at 13:00
0

The member is in RECOVERING since its performing startup self-checks & it is not able to transient into the SECONDARY state since it contains stale data which is due to the oplog size issue.

You could change the size of the oplog using the below steps:

Increasing oplog:

Shutdown the primary server

use admin

db.shutdownServer()

Start primary as standalone & run on different port say 37017

Login to mongo in port 37017

mongo --port 37017

Remove the old contents in local database

For safety have backop of old oplog before dropping

mongodump --db local --collection 'oplog.rs' --port 37017

Drop the old contents in local database

use local

db.oplog.rs.drop()

db.me.drop()

db.replset.election.drop()

db.replset.minvalid.drop()

db.startup_log.drop()

Replset collection cannot be dropped so remove it with required id :

db.system.replset.remove({ "_id" : "your_replsetname"})

Create a new oplog of required size say 50 GB

db.runCommand( { create: "oplog.rs", capped: true, size: (50 * 1024 * 1024 * 1024) } )

Also you can specify the oplog size in MB in mongod.conf file,say for 50 GB its 429496 MB

replication:
   oplogSizeMB: 429496

Hope this Helps !!!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.