We have a MongoDB replica set with 3 machines outside our network and one inside network. The inside one is connected to leader by reverse SSH tunnel such as :
ssh -R 37017:inside_machine:37017 user@leadermachine
Everything works well if the leader doesn't change. When it does, we need to renew the tunnel and hosts entry for the new leader (host entry should resolve the same ip as the leader) but replica set configs don't match anymore,
InvalidReplicaSetConfig: Our replica set configuration is invalid or does not include us
And we need to remove the inside machine, drop local db and readd to replica set. This means the inside machine drops everything and starts to replicate from the very beginning.
While this is not much of a problem for litte db's, ours are large and replication may as well take 2 days.
Are we doing something wrong or is there anything we can do to prevent this?
Thanks a lot.