How to optimize performance of backup process? Servers go down during backup

Question

The context - So I am an intern (So I'll appreciate a more noob friendly response) and we have several websites on a server of which we do a backup daily, at midnight. We use MongoDB and there's a cron job with a script which does a mongodump to a temp dir on the server, the contents of which are then rsynced onto a remote backup server and the temp dir is deleted. We have several websites on there. The database of each is around 2GB, with the exception of two websites with a DB for 4GB and 5GB respectively. In total, all databases are about 23-24GB. There is also a completely separate server with only one website with about 100GB database.

The problem - We monitor our websites using Pingdom and New Relic and each night, during the backups, we get alerts for both servers. The logs don't say much, so I'm guessing they didn't go down completely, just became very very slow. Haven't observed it myself on any of these websites, but I have on a stage website when taking a manual backup. It did come to a crawl.

I am told this didn't used to happen before, even though the way we're backing up hasn't changed, since I came here. I'm guessing, that's because DBs are now larger or there's more of them.

The question - Is there a way to make the whole backup process more performance efficient, so the servers don't go down or come to a crawl?

My ideas - I've been reading about compact and was wondering if you think this would help? I'll probably try it on the stage servers first. Also, is there any way to not write the dump to a temp dir first, but instead to put it directly on the remote server? I'm hoping this would reduce the I/O operations on the live server and improve performance, though I can't seem to find a way to do it.

Some advice will be greatly appreciated, as the project manager is crying about the servers going down at night.

Thanks.

Answer 1

let us discuss few approaches for backup

1. If you are using replSet then could be very easy to take backup from secondary hidden if possible.It will simply makes things easy for you. 3 member replica set 1 primary ,1 secondary ,1 secondary but hidden(can be used for backup)

2. you can use snapshot using LVM ,as it is system based approach so need manual intervention.for details Click here!

3. You can go for MMS(MongoDB Management Service) It will use your oplog to create backup.If not using replset then also you can create OPlog click here for details.

Also have a look at strategies comparison click here

Answer 2

So I ended up solving this issue myself. Not quite in the way I was hoping to do it, but it does work and the websites are no longer going down or even slowing down noticeably during backups.

The fix - I changed the kernel I/O Scheduler from CFQ to Deadline, i.e. made it prioritize smaller, faster I/O tasks. That way any website request is prioritized over the backup process and the websites have no down time anymore. Tested and worked just right.

Here's how to do it. We first check what the scheduler is currently set to.

# cat /sys/block/hda/queue/scheduler
noop deadline [cfq]

Then we can modify it to whatever we want like so

# echo deadline > /sys/block/hda/queue/scheduler

Confirming it's changed

# cat /sys/block/hda/queue/scheduler
noop [deadline] cfq

This may need to be done for more disks than just one, assuming hda isn't your only hard drive. E.g. I also had a hdb disk, so I had to do this

# echo deadline > /sys/block/hda/queue/scheduler

Answer 3

Adding secondary hidden delayed member to a replica set: https://docs.mongodb.org/manual/tutorial/configure-a-delayed-replica-set-member/
At the worst case scenario you will promote that node to a primary and kill others. MongoDB Cloud is great and we use it in production, but it may cost you more and restoring is not straight forward.