Set replica set member host names on Secondary

Question

I have a replica set rset1 that consists of a primary, secondary, and a backup server.

I recently just changed their DNS names in Route53--the only name that didn't change was the Primary's, however since it can't connect to the secondaries, it stepped-down from being Primary.

I've been trying write a query to use in an update of local.system.replset, where local is the database and system.replset the collection; I can't seem to get anywhere with it, though.

Here's what system.replset looks like:

{ "_id" : "rset1",
  "version" : 7,
  "members" : [
    { "_id" : 0,
      "host" : "mongo-rs1-p.foo-bar.com:27017", 
      "priority" : 3
    }, 
    { "_id" : 1,
      "host" : "mongo-rs1-s1.foo-bar.com:27017",
      "priority" : 2
    },
    { "_id" : 2,
      "host" : "mongo-rs1-s0.foo-bar.com:27017"
    }
  ] 
}

I need to change members to this:

 [
   { "_id": 0,
     "host": "mongo-rs-p.foo-bar.com:27017",
     "priority": 3
   },
   { "_id": 1,
     "host": "mongo-rs1-s.foo-bar.com:27017",
     "priority": 2
   },
   { "_id": 2,
     "host": "mongo-rs1-b.foo-bar.com:27017"
   }
]

I've tried using:

cfg = rs.conf()
cfg.members[1].host = "mongo-rs1-s.foo-bar.com:27017"
cfg.members[2].host = "mongo-rs1-b.foo-bar.com:27017"
rs.reconfig(cfg)

But since the primary is no longer primary, I can't get this to run successfully, and running db.getMongo().setSlaveOk() didn't help with using rs.reconfig()

As a last resort (I've had this problem before) I can run rm -rf /data; mkdir -p /data to remove mongo's data directory (after killing the mongod process) and then I can start the mongod server again and rebuild the replica set from scratch, but I'd rather not do that.

So how can I update the members of the replica set without deleting the database on a primary that stepped down?

Answer 1

There is a guide for how to do this (reconfigure a replica set with unavailable members) here:

http://docs.mongodb.org/manual/tutorial/reconfigure-replica-set-with-unavailable-members/

You essentially have to do a forced reconfig of the set because there is no primary available. If you want to keep a set operational while doing this type of move, you can:

1. Use your hosts file to temporarily point the old names to the new (or similar) and "fool" the primary into thinking they are up (then make the permanent changes one at a time)
2. Temporarily add arbiters to prevent the primary from stepping down (2 more for example, so the primary is kept with 3/5 votes)
3. Reconfigure the set to a single node before doing the moves
4. Do the moves one server at a time, ensuring there are 2/3 nodes up to form a majority at all times (and hence keep a primary up)