2

I'm working on a migration of MongoDB Replica Set from on premises to AWS. It's 300 GB of data. I've evaluated the following options:

  1. MongoDB replica sets. It's really slow, initial sync takes 4/5 days.
  2. Rsync. Rsync while MongoDB is running, then stop it, and run Rsync again. It doesn't work as expected, Rsync for some reason always takes the same amount of time.
  3. Rsync + Replica Set. Stop a secondary member, rsync, start mongo, and then wait MongoDB replication to sync the latest updates.

I have questions for the first two options.

  1. I do not understand why initial sync is so slow.
  2. I do not understand why rsync is copying all the data always. I'm using rsync -a.

Option 3 should work, testing it. But if you have better options, it would be great.

1

Normally slow initial sync is because of slow network connection or VERY slow data disks. Have you measured what is transfer speed during initial sync.

Reason, why all data is copied with rsync, is that, that rsync thinks that too much has changed for delta copy.

And, yes option 3 is best solution, but only if you do that copy from snapshot or some other freezed data. Don't do it from live data files what are changing during copy process.

  • I thought about the same, but rsync is 50x faster on the same src and dst machine. So, it's not the network nor the disks. BTW, I've confirmed with md5sum that a lot of big files changed, so option B is not viable. – Diego Woitasen Dec 4 '17 at 1:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.