I have a cluster with 7 nodes: 1 primary, 3 secondary and 3 arbiters. I also have several more app servers which communicate with the cluster. Recently, we've received lots of errors in the mongodb log (prim / secondary) like
Error in heartbeat request to ARBITER01:<port>
Also, from mongodb logs of the 3 arbiters having the same errors of timeout when trying to connect to other arbiters. The loads are high in the 3 arbiters due to we captured lots of CLOSE_WAIT (>300) and SYN_RECV (>100) and most of them pointing to the app servers
What can cause these errors while the app doesn't have anything to do with ARBITERS?
~]# netstat -an | grep <app IP>
tcp 0 1 ::ffff::48098 ::ffff::27757 SYN_SENT
tcp 0 1 ::ffff::44796 ::ffff::27737 SYN_SENT
tcp 0 1 ::ffff::60354 ::ffff::27721 SYN_SENT
tcp 0 1 ::ffff::49100 ::ffff::27719 SYN_SENT
tcp 0 1 ::ffff::52536 ::ffff::27717 SYN_SENT
tcp 0 1 ::ffff:.159:44446 :::27720 SYN_SENT
Is there any bug from java driver used with mongodb 3.2.19?
rpm -qa | grep mongo
The reason we have 3 arbiters is because we have 3 regions for the replica set of 7 nodes. We use VM Centos 7 for all the nodes and the arbiters have 8 CPUs. Is it true that we can reduce the load if all arbiter members are set to priority 0 and hidden = true as we are in MongoDB version 3.2.12 (arbiter's priority default is 0 for 3.6)? The issue here is we see too many CLOSE_WAIT and SYN_RECV sockets with several connections from the app servers. They come and go but the numbers are always high. It's also caused sometime, the primary/secondaries' logs showing heartbeat timeout to these arbiters.