1

I have a cluster with 7 nodes: 1 primary, 3 secondary and 3 arbiters. I also have several more app servers which communicate with the cluster. Recently, we've received lots of errors in the mongodb log (prim / secondary) like

Error in heartbeat request to ARBITER01:<port>

Also, from mongodb logs of the 3 arbiters having the same errors of timeout when trying to connect to other arbiters. The loads are high in the 3 arbiters due to we captured lots of CLOSE_WAIT (>300) and SYN_RECV (>100) and most of them pointing to the app servers

Questions:

  1. What can cause these errors while the app doesn't have anything to do with ARBITERS?

    ~]# netstat -an | grep <app IP>
    

    tcp 0 1 ::ffff::48098 ::ffff::27757 SYN_SENT
    tcp 0 1 ::ffff::44796 ::ffff::27737 SYN_SENT
    tcp 0 1 ::ffff::60354 ::ffff::27721 SYN_SENT
    tcp 0 1 ::ffff::49100 ::ffff::27719 SYN_SENT
    tcp 0 1 ::ffff::52536 ::ffff::27717 SYN_SENT
    tcp 0 1 ::ffff:.159:44446 :::27720 SYN_SENT

  2. Is there any bug from java driver used with mongodb 3.2.19?

    rpm -qa | grep mongo
    

    mongodb-org-server-3.2.19-1.el7.x86_64
    mongodb-org-shell-3.2.19-1.el7.x86_64
    mongodb-org-tools-3.2.19-1.el7.x86_64
    mongodb-org-mongos-3.2.19-1.el7.x86_64
    mongodb-org-3.2.19-1.el7.x86_64

The reason we have 3 arbiters is because we have 3 regions for the replica set of 7 nodes. We use VM Centos 7 for all the nodes and the arbiters have 8 CPUs. Is it true that we can reduce the load if all arbiter members are set to priority 0 and hidden = true as we are in MongoDB version 3.2.12 (arbiter's priority default is 0 for 3.6)? The issue here is we see too many CLOSE_WAIT and SYN_RECV sockets with several connections from the app servers. They come and go but the numbers are always high. It's also caused sometime, the primary/secondaries' logs showing heartbeat timeout to these arbiters.

  • I'm finding this question very difficult to read. Could you perhaps a) use more line-breaks and b) use the formatting tools at the top of the edit window to make the question clearer? I'm afraid that I know next to nothing about MongoDB, but a clearly written question will assist you in getting good answers - help us to help you! p.s. welcome to the forum :-) – Vérace Jun 7 at 4:44
  • How the nodes are split between region? I mean which nodes are in which region? It looks like you don't need three arbiters. Also, you have only 3 regions and why there 4 data nodes(1 primary and 3 secondaries). Please post the rs.status() value as well. – Mani Jun 7 at 21:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.