I get the problem in many past days. I have MongoDB cluster with 3 shards and 3 replicas.
1 2 3
A O S S
B S P S
C P O P
P - Primary state
S - Secoundary state
O - Other state
Alphabets are replica machine and numberic are shards.
I try to resync all data around 2TB in mongod
machine with other state (A1 and C2). But I got error while resync data in primary that mongod service going down because Too many open file
2019-03-16T16:35:22.351+0000 E - [conn28204] cannot open /dev/urandom Too many open files in system
2019-03-16T16:35:22.362+0000 I NETWORK [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files in system
2019-03-16T16:35:22.362+0000 I NETWORK [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files in system
2019-03-16T16:35:22.362+0000 I NETWORK [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files in system
I already try to fixed ulimit as mongodb recommendation.
> ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 241518
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 64000
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 241518
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
and set in /etc/security/limits.conf like this
* soft nofile 64000
* hard nofile 64000
root soft nofile 64000
root hard nofile 64000
but all of this is not fix my problem. I still get mongod service down because Too many open file
. I stuck for 3 days. Anyone have any ideas or solutions?