Mongo DB Replica set Stuck at RECOVERING state

Question

We have created a replica set and now the problem is 2 members of replica set [3 members set] are in recovering mode from 48 hours. Initially the size of recovering nodes was increasing and now even that has stopped. So in recovering nodes they are stuck after 90 GB of data with 60+ GB of local data.

How to come out of this mode ?

Answer 1

The easy, albeit a bit unsecure way

1. Stop the first secondary
2. Delete the content of it's dbpath
3. Restart the secondary
4. Wait for it to catch up with the primary
5. Repeat process with the second secondary

This is a bit unsecure as it is unknown why the secondaries entered the Recovering state.

The more secure, but also more intrusive way

As above, but stop your application during the process. This prevents the possibility that your application is inserting more data than the secondaries are able to replicate. However, the problem may occur during production.

The most secure, but also most intrusive way

1. Shut down the whole replica set
2. Remove the content of dbpath on both secondaries
3. Copy the content of dbpath to both secondaries' dbpath
4. Start the old primary.
5. Start one of the old secondaries.
6. Wait until a new primary is elected.
7. Start the remaining secondary.

Some notes:

Use MMS. It's free, it's easy to set up and it gives you good information about your replica set. Try to keep the value for "replication lag" around 0, and take all means necessary that your replication lag is never greater than the "replication oplog window".

Always make sure you have a 1Gb network and a (sorry) shitload of RAM. The more, the better. Additional rule of thumb: rather half the RAM and SSDs than double the RAM and no SSDs (with RAM remaining within reasonable limits).

Disclaimer: Always make a backup of production data before fiddling with it.

Answer 2

The replication process fails even if you start scratch from a new dbpath on the secondary.So the thing is to make some changes in the oplog. The size of the oplog must be set to a optimal value that so that it should be able to handle all application writes into it.

Increasing oplog size:

Shutdown the primary server

use admin

db.shutdownServer()

Start primary as standalone & run on different port say 37017

Login to mongo in port 37017

mongo --port 37017

Remove the old contents in local database

For safety have backop of old oplog before dropping

mongodump --db local --collection 'oplog.rs' --port 37017

Drop the old contents in local database

use local

db.oplog.rs.drop()

db.me.drop()

db.replset.election.drop()

db.replset.minvalid.drop()

db.startup_log.drop()

Replset collection cannot be dropped so remove it with required id :

db.system.replset.remove({ "_id" : "your_replsetname"})

Create a new oplog of required size say 50 GB

db.runCommand( { create: "oplog.rs", capped: true, size: (50 * 1024 * 1024 * 1024) } )

Also you can specify the oplog size in MB in mongod.conf file,say for 50 GB its 429496 MB

replication:
   oplogSizeMB: 429496

Hope this Helps !!!

Edit:

As mentioned by Nicholas Tolley Cottrell in comments. In the MongoDB version 3.6 we can change oplog size in runtime without restart.

Check current oplog size

use local
db.oplog.rs.stats().maxSize

To change the oplog size to 10 GB

db.adminCommand({replSetResizeOplog: 1, size: 10000})