2

I have a cluster with 7 nodes: 1 primary, 3 secondary and 3 arbiters. I also have several more app servers which communicate with the cluster. Recently, we've received lots of errors in the mongodb log (prim / secondary) like

Error in heartbeat request to ARBITER01:<port>

Also, from mongodb logs of the 3 arbiters having the same errors of timeout when trying to connect to other arbiters. The loads are high in the 3 arbiters due to we captured lots of CLOSE_WAIT (>300) and SYN_RECV (>100) and most of them pointing to the app servers

Questions:

  1. What can cause these errors while the app doesn't have anything to do with ARBITERS?

    ~]# netstat -an | grep <app IP>
    

    tcp 0 1 ::ffff::48098 ::ffff::27757 SYN_SENT
    tcp 0 1 ::ffff::44796 ::ffff::27737 SYN_SENT
    tcp 0 1 ::ffff::60354 ::ffff::27721 SYN_SENT
    tcp 0 1 ::ffff::49100 ::ffff::27719 SYN_SENT
    tcp 0 1 ::ffff::52536 ::ffff::27717 SYN_SENT
    tcp 0 1 ::ffff:.159:44446 :::27720 SYN_SENT

  2. Is there any bug from java driver used with mongodb 3.2.19?

    rpm -qa | grep mongo
    

    mongodb-org-server-3.2.19-1.el7.x86_64
    mongodb-org-shell-3.2.19-1.el7.x86_64
    mongodb-org-tools-3.2.19-1.el7.x86_64
    mongodb-org-mongos-3.2.19-1.el7.x86_64
    mongodb-org-3.2.19-1.el7.x86_64

The reason we have 3 arbiters is because we have 3 regions for the replica set of 7 nodes. We use VM Centos 7 for all the nodes and the arbiters have 8 CPUs. Is it true that we can reduce the load if all arbiter members are set to priority 0 and hidden = true as we are in MongoDB version 3.2.12 (arbiter's priority default is 0 for 3.6)? The issue here is we see too many CLOSE_WAIT and SYN_RECV sockets with several connections from the app servers. They come and go but the numbers are always high. It's also caused sometime, the primary/secondaries' logs showing heartbeat timeout to these arbiters.

2
  • I'm finding this question very difficult to read. Could you perhaps a) use more line-breaks and b) use the formatting tools at the top of the edit window to make the question clearer? I'm afraid that I know next to nothing about MongoDB, but a clearly written question will assist you in getting good answers - help us to help you! p.s. welcome to the forum :-)
    – Vérace
    Jun 7, 2019 at 4:44
  • How the nodes are split between region? I mean which nodes are in which region? It looks like you don't need three arbiters. Also, you have only 3 regions and why there 4 data nodes(1 primary and 3 secondaries). Please post the rs.status() value as well.
    – Mani
    Jun 7, 2019 at 21:45

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.