Is it possible to have 2 arbiter on a replica set on my setup?

Question

I am trying to upgrade mongodb from 4.4 > 5.0

I have a 2 separate replica set (replicaset1, replicaset2) which hosts different platform on our site.

replicaset1 setup: (mongodb version 4.4)
rs1VM1 (10.1.0.1:27017) - PRIMARY
rs1VM2 (10.1.0.2:27017) - SECONDARY
rs2VM1 (10.1.0.10:27018) - ARBITER

replicaset2 setup: (mongodb version 4.4)
rs2VM1 (10.1.0.10:27017) - PRIMARY
rs2VM2 (10.1.0.11:27017) - SECONDARY
rs1VM1 (10.1.0.1:27018) - ARBITER

We happened to put the arbiter for each replicaset on the other platform to provide at least a little bit of HA on our setup.

Now I want to upgrade only the replicaset1 from 4.4 > 5.0

My proposed approach would be:

1. Create a arbiter on rs1VM1 (10.1.0.1:27017) that would be 10.1.0.1:27018 - ARBITER
2. Delete the old arbiter rs2VM1 (10.1.0.10:27018) on the replicaset1
3. Shutdown the secondary then upgrade the binary files. Restart mongod on secondary

After this the setup would be

replicaset1 setup: (mongodb version 4.4)
rs1VM1 (10.1.0.1:27017) - PRIMARY (mongod 4.4)
rs1VM2 (10.1.0.2:27017) - SECONDARY (mongod 5.0)
rs1VM3 (10.1.0.1:27018) - ARBITER (mongod 4.4)

4. Once secondary is already good. I would create another arbiter which will be located on rs1VM2 (10.1.0.2:27018) After this the setup would be

replicaset1 setup: 
rs1VM1 (10.1.0.1:27017) - PRIMARY (mongod 4.4)
rs1VM2 (10.1.0.2:27017) - SECONDARY (mongod 5.0)
rs1VM2 (10.1.0.2:27018) - ARBITER (mongod 5.0)

5. Now I can now upgrade the PRIMARY. Perform rs.stepDown() then db.shutdownServer(), then upgrade the binary files to 5.0 then restart mongodb service

replicaset1 setup: 
rs1VM1 (10.1.0.1:27017) - PRIMARY (mongod 5.0)
rs1VM2 (10.1.0.2:27017) - SECONDARY (mongod 5.0)
rs1VM2 (10.1.0.2:27018) - ARBITER (mongod 5.0)

is this feasible or there are much more easier approach to this

Answer 1

Yes, you can do that. However, because arbiters don't have any DB data, there is no reason to do that. Just upgrade your arbiters to 5.0, then upgrade your secondaries to 5.0, when those secondaries are "up-to-date", step down primaries (they become secondary) and upgrade those last nodes to 5.0.

Look here!

Answer 2

I deviated from my original approach and I tried to download the tgz and not the normal installer for mongodb as per suggestion of @Wernfried which contains the binary files.

From there, the method that I did was pretty simple.

1. Upgrade the Arbiter, Secondary then the Primary

After extracting the tgz file, I just pointed the mongod on the new directory where I extracted the tgz file. (which contains the updated binary mongod, mongo, etc.)

sudo nohup mongod --port 27018--dbpath /datadrive/mongodb-arb --replSet rep1 --bind_ip 0.0.0.0 --keyFile /home/test/repkey &

Then the upgrade process went smooth