We host a MongoDB 3.4 replica set in AWS with three nodes: a primary, secondary failover and an arbiter. Normally, if the primary instance dies, the failover to the secondary is pretty quick (10-30 seconds).
Today we had a network issue where the MongoDB primary instance lost connectivity with the disk containing the database for about 3 minutes and CPU IOWait went to 100%. During this time, queries to the primary just hung and went into timeout. Probably because the primary was still up (though unresponsive), the replica set did not failover or even start a vote.
Is there a configuration which would produce a failover also in such cases? Or is there some ready tools that could force a failover if simple queries to the primary node start taking too long?