MongoDB Too many open file while reSync member

Question

I get the problem in many past days. I have MongoDB cluster with 3 shards and 3 replicas.

    1   2   3
A   O   S   S
B   S   P   S
C   P   O   P

P - Primary state
S - Secoundary state
O - Other state

Alphabets are replica machine and numberic are shards.

I try to resync all data around 2TB in mongod machine with other state (A1 and C2). But I got error while resync data in primary that mongod service going down because Too many open file

2019-03-16T16:35:22.351+0000 E -        [conn28204] cannot open /dev/urandom Too many open files in system
2019-03-16T16:35:22.362+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files in system
2019-03-16T16:35:22.362+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files in system
2019-03-16T16:35:22.362+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files in system

I already try to fixed ulimit as mongodb recommendation.

> ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 241518
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 64000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 241518
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

and set in /etc/security/limits.conf like this

*            soft    nofile      64000
*            hard    nofile      64000
root         soft    nofile      64000
root         hard    nofile      64000

but all of this is not fix my problem. I still get mongod service down because Too many open file. I stuck for 3 days. Anyone have any ideas or solutions?

Answer 1

Since you have 2TB of data to be resynced, try the following approach.

Follow the below approach for each replica set as required.

1. Connect to the Primary and back up the database using mongodump, also take the latest document oplog collection.

2. Shutdown the secondary where you want to resync the data.

3. Start this secondary as standalone server

4. Drop the oplog collection in the local database and recreate it by inserting the latest entry from the primary node.

5. restore the backup data taken from step 1 using mongorestore

6. Once the restoration is completed, shut down and restart the server as replica set member.