My MongoDB replicaset running in cloud and onprem on version 2.6.12. The replication between the two is failing everytime it reaches around 1900 GB.
I have tried the below steps but the replication still fails around 1900 GB. The total data for the replication is around 6 TB.
Relevant logs:
2023-07-27T20:45:08.048+0000 [journal] warning couldn't write to / rename file /var/lib/mongodb/journal/prealloc.0: Couldn't open directory '/var/lib/mongodb/journal' for flushing: errno:24 Too many open files
2023-07-27T20:45:08.048+0000 [journal] User Assertion: 13516:couldn't open file /var/lib/mongodb/journal/j._1345 for writing errno:24 Too many open files
2023-07-27T20:45:08.048+0000 [journal] warning exception opening journal file couldn't open file /var/lib/mongodb/journal/j._1345 for writing errno:24 Too many open files
2023-07-27T20:45:08.048+0000 [journal] error exception in dur::journal couldn't open file /var/lib/mongodb/journal/j._1345 for writing errno:24 Too many open files
2023-07-27T20:45:08.048+0000 [journal] dbexception in groupCommitLL causing immediate shutdown: 13516 couldn't open file /var/lib/mongodb/journal/j._1345 for writing errno:24 Too many open files
2023-07-27T20:45:08.048+0000 [journal] SEVERE: dur1
2023-07-27T20:45:08.054+0000 [journal] SEVERE: Got signal: 6 (Aborted).
My configuration are:
/etc/systemd/system/mongod.service
LimitNoFILE=infinity
LimitFSIZE=infinity
/etc/sysctl.conf
fs.file-max = 999899999
fs.nr_open = 999999999
/etc/security/limits.conf
* soft nofile 6047500
* hard nofile 6047500
mongodb soft nofile 6047500
mongodb hard nofile 6047500
ulimit -n shows 6047500.
I was wondering about what else could be done to fix the problem.
