5

I'm trying to archive something like this enter image description here

The MongoDB instance on dc1.com is currently a standalone, and On-Site periodically pulls a backup via mongodump over a ssh tunnel.

I now want to create two instances, one On-Site and one in a second datacenter, and make the three instances become a replica set, so that the backup is almost instant. This way I would have an On-Site backup and one in the second datacenter, which would always be up to date.

If the main MongoDB instance in dc1.com fails, then that's it, no data can be served, this will then need to get fixed manually, as the On-Site instance and the one in the secondary datacenter should not take over. Nor are those two meant to be queried. They are there just as a backup.

My problem is that when I add --replSet to the main database, and then a rs.initialize(), that this instance is then somehow not recognized as bound to localhost, but to 172.17.0.1, which is the docker interface, which I need to bind to so that the containers can connect to MongoDB. If I then try to add a member via rs.add("localhost:2001"), which is the instance on dc2.com, I get an error about not being able to mix localhost and non-localhost databases. The main instance is recognized as 172.17.0.1 instead as localhost.

"Either all host names in a replica set configuration must be localhost references, or none must be; found 1 out of 2"

I then used https://stackoverflow.com/questions/28843496/cant-initiate-replica-set-in-ubuntu in order to issue a rs.initiate({_id:"rs0", members: [{"_id":1, "host":"127.0.0.1:1000"}]}) to force the initialization of the replica set on dc1.com to be bound to localhost. But when I then try to add a member via rs.add("localhost:2001") I still am not able to do it. It's no longer the hostname error, but another one, which I can't quite remember, as I got tired of trying and had to rollback everything. It was something like connection rejected. When I do a mongo --port 2001 I get disconnected after a warning about something related to "isMaster".

Is it possible to create this kind of setup? All I'm trying to do is to avoid using TLS on MongoDB and binding the backups to globally accessible interfaces (the On-Site instance would be behind a firewall, so 0.0.0.0:1002 would not be publicly accessible)

3
  • Your idea of putting each mongod on a different port, and then adding each one to the replica set with the address 127.0.0.1, looks like a good one. Can you confirm that this definitely isn't working? May 18, 2017 at 12:18
  • I will try this again in a couple of hours. Maybe it's an issue that the port mapping is asymetric (2001 on origin connects to 1001 on remote), and that remote can't initiate a connection back to the primary and the other secondary. Maybe it's also required that the containers be able to talk to the secondaries, which they can't, since they don't have access to the host's localhost. I'm considering TLS atm, but I'd prefer this SSH solution. There's also a thing in {_id:"rs0", members: [{"_id":1, "host":"127.0.0.1:1000"}]}, where _id maybe needs to be 0.
    – Daniel F
    May 18, 2017 at 12:22
  • Ah yes, those would cause problems. MongoDB replica sets can only work if every node can contact every other node, by the address and port in the replica set configuration; so your port mapping needs to be symmetric, and you need a tunnel between every pair of nodes. That might end up being more effort than it's worth, I fear. May 18, 2017 at 12:34

2 Answers 2

3

The following setup is possible, without using TLS.

Datacenter 1 is the primary datacenter. All clients connect only to this one. Datacenter 2 is only for realtime backups via replication. On-Site is also only for realtime backups via replication.

This setup is NOT meant for failover, only for disaster recovery.

Datacenter 1:

MongoDB

mongod --bind_ip 127.0.0.1,172.17.0.1 --replSet test --port 2000 --rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger --wiredTigerDirectoryForIndexes --fork

127.0.0.1 is for SSH access, 172.17.0.1 is for access from Docker containers.

SSH Tunnels (this server is responsible for creating the tunnels to Datacenter 2):

autossh -f -N -L 2001:localhost:2001 [email protected]

autossh -f -N -R 2000:localhost:2000 [email protected]

Datacenter 2:

MongoDB

mongod --bind_ip 127.0.0.1 --port 2001 --replSet test --rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger --wiredTigerDirectoryForIndexes --fork

On-Site:

MongoDB

mongod --bind_ip 127.0.0.1 --port 2002 --replSet test --rest --httpinterface --logpath data/replica/logs/log.txt --dbpath data/replica/wiredTiger --directoryperdb --storageEngine wiredTiger --wiredTigerDirectoryForIndexes --fork

SSH Tunnels (On-Site is responsible for creating the tunnels to Datacenter 1, no persistent connection exists between Datacenter 2 and On-Site)

autossh -f -N -R 2002:localhost:2002 [email protected]

autossh -f -N -L 2000:localhost:2000 [email protected]


The server in Datacenter 1 contains all the data which is to be replicated, Datacenter 2 and On-Site are empty databases.

On the machine in Datacenter 1 I enter mongo --port 2000, then I issue a

rs.initiate(
   {
      _id: "test",
      version: 1,
      members: [
         { _id: 0, host : "localhost:2000", priority: 1, votes: 1 },
         { _id: 1, host : "localhost:2001", priority: 0, votes: 0, hidden: true },
         { _id: 2, host : "localhost:2002", priority: 0, votes: 0, hidden: true }
      ]
   }
)

I guess priority: 0, votes: 0, hidden: true on the secondaries is optional here, it does what I want, just have the two secondaries be a part as silent, invisible data collectors for backup purposes. I'm not sure if changing this would have any side-effects as it could be possible that then a connection between both secondaries should be available, which it is not, in this case (they can't see each other).

I could use a slaveDelay on one of the hidden members in order to make it lag for let's say half a day, but since I will be making plenty of backups from both hidden databases via mongodump, it's not really neccessary.

In order to be able to connect to the secondaries via mongodb tools I needed to issue a rs.slaveOk()

Nothing had to be changed in the clients, they connect to 172.17.0.1:2000.

I tested a reboot of the machine in Datacenter 2 and during the reboot I inserted new items in the DB, after about 30 seconds the rebooted machine synced to Datacenter 1 as expected.

Datacenter 1 has a fast server with lots of RAM and CPU and only a little bit of SSD storage, Datacenter 2 has a little bit of RAM but lots of HDD, so this seems to be a good solution.

1

Solution is TLS/SSL what is supported in community version too. Without .pem file you cannot login, traffic between nodes is encrypted.. No need for SSH tunnel anymore.

How to configure ssl

Then if only concern is that, that nobody "else" could connect your server, but traffic between servers don't need to be secure, you can use keyfile, without that file (6 to 1024 characters) nobody can connect the server without right credentials.

7
  • Thanks. Is it possible to use TLS selectively per bound interface? There's no need for the docker containers which reside on the same host as the primary to communicate over TLS.
    – Daniel F
    May 18, 2017 at 11:06
  • No. It's not possible. Everything is encrypted and you cannot connect without certificate.
    – JJussi
    May 18, 2017 at 12:02
  • Reading this allanbank.com/blog/security/tls/x.509/2014/10/13/… made me wonder if it is possible to use TLS only for clustering and not for client communication. Because he differentiates among cluser certificates and client certificates, and the YAML file differentiates between PEMKeyFile and clusterFile. Would you happen to know anything about this?
    – Daniel F
    May 18, 2017 at 15:28
  • I was wrong. It's possible to client to connect without right certificate, if you configure server with net.ssl.allowConnectionsWithoutCertificates.
    – JJussi
    May 18, 2017 at 17:18
  • added keyfile option to answer too..
    – JJussi
    May 18, 2017 at 17:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.