I have a replica-set which consists of 3 amazon ec2 nodes : primary, secondary and an arbiter the data volume size is ~400GB and currently it is 90% full. i need to resize the volumes to a more appropriate size. I'm looking for the best solution to do it with minimum downtime/no-downtime and minimum resource overhead - meaning use the least amount of CPU/IO/network needed for the resize process. time is not a big issue as i believe it wouldn't take more than 24 hours.
Here are the logical steps i have come up with so far:
- create, attach & mount new volume for secondary
- stop mongod process on secondary
- copy data from old volume to new volume
- restart mongod process on secondary with new dbpath directing to the new volume OR replace mounting points for old and new volumes, and then restart mongod process
- ensure secondary is finished syncing the remaining data
- create, attach & mount new volume for primary
- stepdown primary
- ensure all processes bounced correctly to the new primary
- repeat 2-5 (on the current secondary)
- [optional] stepdown primary - return to original setup
a) when copying files from one volume to another, assuming the bigger one is also faster (more PIOPS), is it efficient to use parallelism ? as in copy a maximum of ~10 files at a time?
b) some would say if i already put the effort for copying the data then i should simply wipe the data and let it resync anew - however i'm worried about the network and read impacts on the "living" primary which may degrade performance for my applications. is that a right assumption?
c) how can i measure/verify that the oplog retention is sufficient for resyncing the database after copying the files?
d) is there a better idea/solution ?
mongo version - 3.0.10 storage engine - mmapv1 os - amazon linux journaling is on