2

I'm upgrading a secondary replica set member to wiredTiger. I have upgraded it from MongoDB 2.6.3 to 3.0.4 and changed the storage engine to wiredTiger. Now it is resyncing all data from the primary. At some point the following error is received, and the process starts all over again:

2015-07-22T13:18:55.658+0000 I INDEX [rsSync] building index using bulk method

2015-07-22T13:18:55.664+0000 I INDEX [rsSync] build index done. scanned 1591 total records. 0 secs

2015-07-22T13:18:56.397+0000 E STORAGE [rsSync] WiredTiger (24) [1437571136:397083][20413:0x7f3d9ed29700], file:WiredTiger.wt, session.create: WiredTiger.turtle: fopen: Too many open files

2015-07-22T13:18:56.463+0000 E REPL [rsSync] 8 24: Too many open files

2015-07-22T13:18:56.463+0000 E REPL [rsSync] initial sync attempt failed, 9 attempts remaining

The same machine was previously running 2.6.3 version without any open file limits issues. I'm aware that wiredTiger might be creating much more files, so it must be it, but does it keep them all open simultaneously ?

For reference:

cat /proc/sys/fs/file-max

10747371

In /etc/init.d/mongod the configuration is:

ulimit -n 64000

According to the documentation it seems that mongo holds a file descriptor for every data file. As in wiredTiger this results in a file for each collection + a file for each index, according to a calculation for our usecase, can add up to over 700K.

So I can change the ulimit to 700000 or higher, but I'm wondering whether this is the most correct solution, and what alternatives exist.

  • Not sure what your question really is. How to change ulimit? How to make MongoDB open fewer files? – mustaccio Jul 23 '15 at 20:00
  • According to the documentation it seems that mongo holds a file descriptor for every data file. As in wiredTiger this results in a file for each collection + a file for each index, according to a calculation I made, this can add up to over 700K. So I can change the ulimit to 700000, but I'm wondering whether this is the most correct solution. – Baruch Oxman Jul 23 '15 at 20:08
  • I don't think a solution can be most correct: it is either correct or not. I can see a couple of alternative solutions that would also be correct, in that they would resolve EMFILE. You could reduce the number of collections and indexes in your database, or you could also rewrite the engine, it being open source. – mustaccio Jul 23 '15 at 20:20
  • Actually, the most viable options appears to be to not be using wiredTiger... – Baruch Oxman Jul 23 '15 at 20:49
  • Did you ever solve this issue? Probably your use case has too many databases/collections for WiredTiger to handle properly. Did you revert back to MMAP? I was reading an article about RocksDB (blog.parse.com/learn/engineering/…) and they mentioned this issue. – markz May 11 '16 at 17:48
0
  • Most Unix like operating system prevents a single user from using too many system resources.
  • These limits are generally applied using ulimits. Given the ulimits, the default low values can cause the number of issues in MongoDB’s normal course of operations.
  • Every deployment may have unique requirements and settings. MongoDB sets the total number of open files to 64000 by default.
  • But when your deployment runs out of file descriptors, mongodb stops with following error: 2017-01-22T21:15:29.824+0000 E REPL [rsSync] 8 24: Too many open files

Solution:

Since MongoDB 3.0, it is shipped with Pluggable Storage Engine. With WiredTiger as a storage engine, you need to revise your open files value as it uses 1 file descriptor for collection and 1 file descriptor for every index. In multi-tenant applications this can be roughly calculated as assuming:

Total Number of Tenants(T)= 1200
Average number of collection by each Tenant(C)= 10
Average number of indexes per collection(I)= 8

Total number of open files(F)= (T*C) + T(C*I)
F = (1200*10) +1200(80)
F= 108000
This hypothetical use case may require ~108000 file descriptors.

In reality, only your own testing within your specific use case can reveal if there will be any problem with the hardware that you have with these many open files. Set these values in /etc/security/limits.conf You may also need to change nproc according to nofile. nproc should be 50% of nofile for mongodb

0

checkout your /etc/security/limits.conf file and change soft and hard limits according to your need.

* soft nofile 4096
* hard nofile 4096

in the first column * means this limit is applied for all users except root user. you also need to edit /etc/pam.d/common-session* file add the following line in the end :

session required pam_limits.so

or else

ulimit -n 65535 (and no reboot required)

Finally, logout/reboot .

  • How is this different from what we're already doing ? The main concern is that mongo is opening too many file handles in the first place... – Baruch Oxman Aug 3 '16 at 9:24
-1

From http://docs.mongodb.org/manual/reference/ulimit/ :

limit fsize unlimited unlimited  # (file size)
limit cpu unlimited unlimited    # (cpu time)
limit as unlimited unlimited     # (virtual memory size)
limit nofile 64000 64000         # (open files)
limit nproc 64000 64000          # (processes/threads)
  • Please note that I already have this configuration, but the 64000 open files limit is not enough... – Baruch Oxman Jul 24 '15 at 7:42
-1

You stated reaching ~700k files, assuming 2-4 indexes per collection => it amounts to over 100k collections.

It's quite high... Unless we're looking at a multi tenant application, separated per tenant, is this the case? Could you describe your data model?

Note, most modern servers could easily handle a million open file handlers - not optimal but definitely not a show stopper.

To reduce the number of open files you'll need to change the model - not always a trivial feat.

-2

For those using Mongodb on Ubuntu 16.04 with systemd make sure to follow @sumit but you should also edit /etc/systemd/system/multi-user.target.wants/mongodb-01.service file and adding the following lines beneath [Service] directive:

# (file size)
LimitFSIZE=infinity
# (cpu time)
LimitCPU=infinity
# (virtual memory size)
LimitAS=infinity
# (open files)
LimitNOFILE=990000
# (processes/threads)
LimitNPROC=495000

Remember to restart the service or the server.

  • This file does not exist in Ubuntu 16. cat: /etc/systemd/system/multi-user.target.wants/mongodb-01.service: No such file or directory If I add this file, then it prevent Mongo from starting altogether. – Cerin Feb 6 '18 at 18:34
  • I don't know but I had the file already here, I am using Ubuntu 16.04.3 LTS with Mongodb 3.2.18 – sys0dm1n Feb 14 '18 at 15:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.