2

We host a MongoDB 3.4 replica set in AWS with three nodes: a primary, secondary failover and an arbiter. Normally, if the primary instance dies, the failover to the secondary is pretty quick (10-30 seconds).

Today we had a network issue where the MongoDB primary instance lost connectivity with the disk containing the database for about 3 minutes and CPU IOWait went to 100%. During this time, queries to the primary just hung and went into timeout. Probably because the primary was still up (though unresponsive), the replica set did not failover or even start a vote.

Is there a configuration which would produce a failover also in such cases? Or is there some ready tools that could force a failover if simple queries to the primary node start taking too long?

Improve this question

2 Answers 2

Reset to default
4

A closely related question is discussed extensively in the comments on SERVER-14139, a bug report against filed against mongodb. To summarize, it is not feasible to build a fully general hang detection system inside a server process.

The commentary in discuss a monitoring approach that kills either the process or shuts down the operating system and can be done using a cron job or the watchdog daemon. Because a mongod process cannot win an election before it has read and written some data to its storage engine, it is safe to immediately attempt to restart mongod after you kill it. The restarted process should not accept connections and certainly will not be able to win an election for primary.

A ticket linked to SERVER-14139 covers an implementation in the enterprise (non-free) version of MongoDB of a storage watchdog timer. Organizations that can instead use the watchdog daemon or an external monitoring process should prefer that approach, because it can protect against more kinds of resource failure.

Improve this answer
3
  • So the only way to protect against IO problems bringing a whole cluster down is to either hack a watchdog script yourself or buy the enterprise version. Really shakes your faith in MongoDB. :-(
    – Sampo
    Commented Sep 21, 2017 at 17:24
  • 1
    Perhaps I wasn't clear in my answer; I actually believe the watchdog script is a superior approach to the implementation available in the enterprise version. I would recommend the watchdog script or other out-of-process failure detection even to users of the enterprise version. It's straightforward to implement, detects other mongodb failures, and can also be used to catch non-mongodb failures.
    – Andy S
    Commented Sep 21, 2017 at 19:23
  • That's possible, but why isn't there a tried-and-tested watchdog agent shipped by default and instructed to be setup as part of a cluster setup, instead of having everyone learn the hard way that a single IO failure can bring down a whole cluster, search for answers in bug repositories, and have everyone build, test and maintain their own scripts for the purpose?
    – Sampo
    Commented Sep 22, 2017 at 5:43
-1

Normally at primary, you give command rs.stepDown(), but I think that this case it will not work because your secondary cannot read primary opLog.

The fastest thing to do in this situation is bringing firewall between those nodes. So, you give a command that port 27017 (or what ever you use) is blocked. Your secondary and arbiter cannot get the heart beat and they vote that primary must be moved.

Improve this answer
2
  • I'm looking for an automated solution, not something with manual intervention. For manual intervention, isn't there any way to inform a node being unhealthy / force stepdown from the secondary?
    – Sampo
    Commented Sep 4, 2017 at 10:00
  • How about cronjob script where you use timeout -command. timeout -k 2s 1 /bin/cat /var/log/syslog.1 > /dev/null; if [[ $? -gt 0 ]]; then echo "FW rule";fi, you know that reading disk is impossible and you should rise firewall...
    – JJussi
    Commented Sep 4, 2017 at 13:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.