2

We have a MEAN application with a 3-member MongoDB replica set. Somehow the total connection count keeps increasing until the primary is not able to take any new connections and goes down, making the application painfully slow. I have set the ulimit to 30000 and restarted the MongoDB and in no time it got 28000+ connection opened and primary went down.

Any idea why this may be happening? Most of the connections are between the different ports on primary with port 27017 on primary (for instance, primary_server:45002 to primary_server:27017).

  • Please post MongoDB driver and server version too. Also look at MongoDB log to find out any problem. – Saleem Mar 21 '16 at 5:26
1
  1. Make sure you have right UNIX ulimit Settings.You can see also: https://docs.mongodb.org/manual/reference/ulimit/
  2. Use top command to check if the performance bottleneck is CPU overload.Check the mongodb log file if there is any slow query,and you can use MTools to analyse it.Then you need optimize the slow query.
  3. Use free command to check if not enough physical memory is available.If MongoDB has too many connections,you need to check Stack settings.every connection is a thread,need a Stack,Stack settings usually is too large by default in Linux: shell> ulimit -a | grep stack You can see the real used Stack size,like this: shell> cat /proc/$(pidof mongod)/limits | grep stack | awk -F 'size' '{print int($NF)/1024}' The total mem consumed by connections is so large,I suggest you set it smaller,such as 1024: shell> ulimit -s 1024
  • There is nothing going on the server as top shows a load average of 0.02 and there is no query running. Setting up the ulimit is also not helping as whatever limit you set, it will open that many connection(mostly from primary to primary) and take the primary down. – Vinayak Pandey Mar 21 '16 at 4:50
  • we are using mongodb 3.2 along with mongoose 4.03 and on one of the secondary server, netstat | grep 27017 | wc -l gives 13774 and this number is keep increasing by 8-10 every second(same is the number in the mongo log on primary server). I changed the sync target also, but still nothing changed and CPU load alos keep increasing with top output as .66 however RAM usgae doesnt increase at all – Vinayak Pandey Mar 21 '16 at 6:38
  • Perhaps you could see about MongoDB Connection Pooling: blog.mlab.com/2013/11/deep-dive-into-connection-pooling – ryanxu Mar 21 '16 at 7:04
  • But RAM usage is not high, it remains at 3 GB for most of the time(event when connection count shows 10000+ or 25000+) and it's always only one secondary that has that many connections to the primary while the other secondary will have around 300 connections at max. – Vinayak Pandey Mar 21 '16 at 9:36
  • Can this be due to isMaster() being run by the mongoose client? How can I check what these connections are doing? – Vinayak Pandey Mar 21 '16 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.