How is it possible to convert a 3-node replica set to a 1-node replica set during low-usage hours

Question

We have a chat application that runs on MongoDB. For the amount of users that we currently have, it is sufficient to have a 2 node+Arbiter set up during peak hours.

However, during off-peak hours, we would like to turn off the secondary node and the arbiter to save on cloud computing costs.

Is this possible? Simply shutting down the secondary+arbiter makes Mongo unable to elect a new primary and the app fails.

If the best/only solution is to remove the secondary and arbiter from the replica set, what are the consequences of doing when adding them back on the next peak hours cycle?

Answer 1

MongoDB replication is to provide automatic fail over during disaster recovery, minimize downtime during maintenance, to use the secondary for analytical use cases and so on. In your scenario, you don't want to avail these benefits during off-peak hours and as far as I know there is no perfect workaround to do these tasks without downtime.

Simply, shutting down the secondary + arbiter makes Mongo unable to elect a new primary and the app fails

Yes, election process will not happen in a replica set with single node. It needs at least two mongod nodes which might not be present in your case if you remove secondary and an arbiter.

If the best/only solution is to remove the secondary and arbiter from the replica set

You can remove the secondary and arbiter from the replica set by following the steps in this link mongo-replica-set-to-single-server.

what are the consequences of doing when adding them back on the next peak hours cycle?

While adding back the replica sets, you will have to restart mongod instances, issue rs.reconfig() method and it can force the current primary to step down which causes an election. During this window time, your app will not read/write anything from/into database which can last up to 10-30 seconds (for re-configuring replicas), another 20-30 seconds (for restarting mongod instances) based on network bandwidth and replica set configuration. Hence, it is not recommended during peak hours.

Answer 2

You have a misconception here. Replica sets, albeit you can distribute some read load, are not mongodb means of load distribution. That would be a sharded cluster.

As @hashavmb has explained in detail, a replica set is there to provide enhanced accessibility. A rough analogy would be like this

If a MongoDB installation was a computer, a replica set would be a RAID, while the individual shards of a sharded cluster would be CPUs.

This analogy is note quite correct, since shards can (and often are used) to distribute IOPS amongst multiple machines, but I think it is sufficient in this case.

Both replica sets and sharded clusters are not meant to be configured dynamically.

Excursus: While it would be possible to remove a shard dynamically, the time and resources needed to move all data from the removed shard to the remaining ones make it a prohibitively expensive procedure. For very large shards draining one of data can take days, as only one chunk of 64MB size will be moved at a time and all other operations have precedence.

The first problem you would encounter when trying to dynamically configure a replica set is that the configuration for said set is replicated to all data bearing nodes. So, in order to make sure to have the least downtime possible, you'd have to restart a single data bearing node – the one with the newest data preferably (remember eventual consistency). That is your primary. But you can not simply restart that, because your driver connected to a replica set would notice that there is another node elected to primary and simply write to that node – and here is where the problem starts. The old primary (the one you are about to remove from the replica set) would not have the data written to the new primary and as soon as removed from the replica set have no means to acquire that data. And if the replica set is put back online, the both nodes would have documents created after their last synchronization. Side note: A MongoDB replica set can notice such split brain situations and rolls back the data inconsistent between the nodes. The data removed is even stored away. But it needs manually interaction – and not a small one – to apply that rolled back data correctly to prevent loss of data.

Ok, fine, so obviously we first should shut down the application before reconfiguring the replica set. Since after doing so no new writes happen, we can simply shut down the nodes of the replica set and restart the old primary in standalone mode by omitting the --replSet command line option or commenting out replSetName in the config file. This could be achieved via cron scripts:

1. T+0: shut down application
2. T+1: Shut down all replica set members, change the config file of primary
3. T+2: Start old primary
4. T+3: Restart application

We'd have to have the application down for 4 minutes+ since cron only has minute precision and we are talking of synchronizing a rather complex process among multiple machines. While we could run a centralized script, I'd advice against that: What happens if it fails after step 1?

Now, just before your high load hours, we have to reverse the process. Sounds easy, right? Except it is not.

You'd need the data written to the "single" MongoDB instance replicated to the other node(s). Replication is done via a special collection called oplog. The first problem is that this collection by default is not updated for standalone instances, but you can change this behavior. The second problem is that the oplog has a fixed size for various reasons. If more operations are logged than the oplog can hold, the oldest operations are removed. So when the oldest operation the other members of a replica set know about is removed, they will notice that and a resync takes place, during which all data will be copied over from the primary to the secondary. During this time, network traffic and disk I/O is drastically increased, and the replica set has no failover capabilities. And, depending on your data size, this resync might take quite a while. So, in order to make sure you have your resync done when your high load time starts, you need to start it early. Which somehow defies what you are trying to achieve the first place.

So, let's see what this means overall. You'd end up with

1. a rather complex procedure (high complexity enhances the probability that something goes wrong),
2. which needs to be synchronized on at least 3 nodes (what happens if one of your ntpds is off for some reason and the MongoDB instances are shut down before the application?), with
3. a guaranteed downtime of at least 4 minutes per configuration change or 8 minutes to change forth and back.

Since the procedure will affect your core services, you'd need to make the scripts as robust as possible, which translates to longer development time, pretty complex fail tests and development of failure handling, all of which in turn translates to money. A lot of it. Even when you calculate at the lower bounds, including taxes and everything a reasonable developer or sysadmin is no less than $500 a day (I know colleagues which charge a lot more). Of the top of my head, I'd say a good developer and a good sysadmin would need about two weeks to make this work and have it tested and documented properly. Summing 10 grands, you can have quite some machines run quite a time – without the drawbacks of downtime or enhanced risks.

For availability calculations 8 minutes would be 0.0015% of a year – do this procedure just 7 times a year, and it would be impossible to have an availability of 99.999%. Doing this once every day would reduce the availability to 99.5%. Other downtimes not counted. I doubt that this is acceptable for any serious business except under very special circumstances.

If you ask me: If money is so tight that a second data bearing node and an arbiter need to be shut down whenever possible, I would not use a replica set in the first place.