I created a 3 nodes replicaset with mongodb that I host in a swarm environment. To do so, I created 3 services mongo1, mongo2, mongo3 with the same configuration.
When I challenged the resiliency of the system by turning of one node e.g.:
docker service scale mongo_mongo2=0
The cluster became very slow. I found that this is caused by the DNS resolution. In swarm, it seems that when the service is down the ip associated to the service is deallocated, hence the DNS service is swarm takes about 10 seconds before replying an error. This long timeout causes the other mongo node to be very slow.
Does anyone have any idea how to handle that case ?
I tried to reduce the timeout, in
options ndots:0 timeout:1
It did get better but it's still very slow.