0

Recently we had a machine physically destroyed, so our replicaset is now: Primary - Secondary - Arbiter (250G on disk, 20G (70hrs) oplog).

So, I'm trying to add a new member (4.4.10) to the replicaset (4.2.5), ( db.adminCommand( { setFeatureCompatibilityVersion: "4.2" } ) has been already done ) but when I'm adding it, the primary simply... crash after 2-3 minutes.

The machine load grows to 4k ! (only load, not CPU, not Memory)

We didn't have a downtime because the secondary became primary and after mongod services restart, the original primary took its place again.

The machines don't have any snapshot / lvm / zfs system available, so i can't take a snapshot.

My next attempt should be to do an rs.syncFrom('[secondary]') quickly after the rs.add({ host:'[new secondary]', priority: 0, votes: 0 }). But if the primary crash and then the rs.syncfrom() make a new crash on the last machine, it will be a bad moment for us.

Is it possible to specify a 'syncFrom' info at the same moment of adding the new member ? (Or any other solution)

Thanks a lot =)

4
  • Do you have different mongo version on your machines? Dec 2 '21 at 22:10
  • @WernfriedDomscheit the actuals machines (primary, secondary, arbiter) are in 4.2.5. The new member is 4.4.10, but I can change it to 4.2.5 and do upgrade later. The objective is to replace all actual machines in 4.2.5 with new machines in 4.4.10
    – Balbuzar
    Dec 3 '21 at 14:17
  • Did you have a look at the mongod logfiles? Dec 3 '21 at 20:51
  • @WernfriedDomscheit Only on the crash timestamp : 2021-12-01T16:08:03.007+0000 E - [ftdc] Assertion: Location13538: couldn't open [/proc/2269/stat] Too many open files src/mongo/util/processinfo_linux.cpp 76 I didn't saw any error during the load increase
    – Balbuzar
    Dec 6 '21 at 16:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.