I have a Mongo Cluster. It's sharded on 6 servers, with 6 replica sets, each replica set having one Primary, and two Secondaries. Additionally, there are 3 config servers. Though, I think the Mongo Cluster configuration is irrelevant to this question.
I have many application servers (currently 100), with the intention of having many more as my volume grows. On each application server, I have a mongos process, 8 http servers (node.js) each bound to a port (using HAProxy as a reverse proxy to route traffic from port 80). Each http server connects to the local mongos process using connection pooling with a poolSize of 3. Pretty simple so far.
The issue is that the number of connections seems to quite simply be:
(number of servers) * (number of http processes) * (poolSize)
In my current case that would be:
100 * 8 * 3 = 2400 total connections
So it would seem that the number of connections scales linearly with the number of servers I'm running. If I scale up 5x, then the number of connections goes to 9,600 which is dangerously close to the 10,000 limit (as well as seeming unnecessarily wasteful). What about a 10x or 100x scale up?
It is stated that "any connection count greater than 1000 – 1500 connections should raise an eyebrow". How can I possibly maintain that number of connections with 100s of application servers? Is a mongos process per application server at this size still appropriate?