Mongo connection failing with 'Transport endpoint is not connected'

Question

I have a 2 server installation.

A web server with Apache and a DB server with MongoDB.

I am load testing it, and on ~300 RPS I am getting this error:

PHP Fatal error: Uncaught exception 'MongoConnectionException' with message 'Transport endpoint is not connected'.

The only thing I am noticing is that right before the fail, I am getting a lots of connections on Mongo:

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
     0    659      0      0       0       1       0   208m  1.28g    40m      0        0          0       0|0     0|0    62k   217k   486   03:57:20 

Almost 500 connections here... but never more than that!

Mongo is 2.0.3. PHP is 5.3.x (latest of Debian install...)

Help!

Answer 1

I'd really need to see the mongod logs to be sure here, but the two most common causes are covered by doing 2 things:

1. Check your ulimits on the client and the server (the too many open files page is a good start)
2. Lower your tcp_keepalive_time from the default (again client and server)

For load testing you should just set the ulimit settings (-n -u etc.) to anything over 20,000 - that is the limit internally in MongoDB currently for open file descriptors. You may also want to make sure that the Apache user you are running under is similarly OK in terms of its ulimit settings give that you are using PHP.

Looking at the output of lsof should give you clues as to where the open file descriptors are coming from.