0

I am trying on add a 3.6 mongoDB node to a 3 member replica set running 3.4. Doing this as a part of DB upgrade from 3.4 to 3.6. When I add the new member to replica set, it goes to STARTUP2, data copy happens for a while at then crashes with following message:

2019-01-16T09:43:57.129+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2019-01-16T09:43:57.129+0000 E -        [conn3505] cannot open /dev/urandom Too many open files
2019-01-16T09:43:57.129+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2019-01-16T09:43:57.129+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2019-01-16T09:43:57.129+0000 F -        [conn3505] Fatal Assertion 28839 at src/mongo/platform/random.cpp 161
2019-01-16T09:43:57.129+0000 I NETWORK  [listener] Error accepting new connection on 0.0.0.0:27017: Too many open files
2019-01-16T09:43:57.129+0000 F -        [conn3505] 

***aborting after fassert() failure

While this is a new mongoDB node, the production servers dont directly send any hits to this machine. Still I see this machine getting lot of hits (way more than the current primary):

2019-01-16T09:51:34.200+0000 I ACCESS   [conn210] Successfully authenticated as principal liveuser on live
2019-01-16T09:51:34.201+0000 I NETWORK  [conn210] end connection XX.XX.XX.XX:36454 (4 connections now open)
2019-01-16T09:51:35.116+0000 I NETWORK  [listener] connection accepted from XX.XX.XX.XX:36456 #211 (5 connections now open)
2019-01-16T09:51:35.116+0000 I NETWORK  [conn211] received client metadata from XX.XX.XX.XX:36456 conn211: { driver: { name: "nodejs", version: "2.2.36" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "4.4.0-1074-aws" }, platform: "Node.js v9.5.0, LE, mongodb-core: 2.1.20" }

There are a lot of these messages which I see in the logs are eventually leading to the above error. My doubt is since this machine is not even secondary yet, is still in STARTUP2, why should this machine receive so many hits? Also, the actual primary doesnt creates so many connections because of connection pooling.

Also, I doubt this is a keyFile issue, since the replication happens till the time it actually crashes. Following is the snippet from my config file, to show my config for bindIp:

storage:
  dbPath: /data/db
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 127.0.0.1,PRI.VAT.E.IP


#processManagement:

security:
  keyFile: /home/ubuntu/mongoKey/keyFile

Due to this I am unable to add any secondary to a replica set. Any idea folks? Thank you.

Edit: Output of ulimit -a:

ubuntu@ip:~/mongoKey$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 983354
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 983354
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Anyways, even if we change this, not sure this will solve the root cause of this issue. I want to understand why this may happen.

0

Recommended ulimit Settings

Every deployment may have unique requirements and settings; however, the following thresholds and settings are particularly important for mongod and mongos deployments:

-f (file size): unlimited
-t (cpu time): unlimited
-v (virtual memory): unlimited [1]
-l (locked-in-memory size): unlimited
-n (open files): 64000
-m (memory size): unlimited [1] [2]
-u (processes/threads): 64000

Your limit for open files is only 1024. I would first change it.

  • edited the post, please have a look – Jeet Kumar Jan 16 at 13:49
  • @Jeet answer edited – Sy10100 Jan 16 at 14:27
  • updated the limits, but still the same issue. The problem lies in the replication. Since the moment I add (do rs.add("ip address")) , the new machine starts getting connection requests, which should not be the case, untill it is up and has become primary. Maybe its the nodejs driver, cant say for sure. – Jeet Kumar Jan 16 at 18:23
  • @JeetKumar do you still have "too many open files" error. What is your error message now ? – Sy10100 Jan 17 at 9:05
  • I still had the same error message, probably because of large size of database. Nevertheless, I worked around the connection requests issue on STARTUP2 state replica set via AWS security groups; disabled all connections to the machine ,except the primary it is syncing to. – Jeet Kumar Jan 18 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.