4

Short version: Is there a way to add a new SECONDARY server to an existing production MongoDB replica set without downtime, and without temporarily disabling the "auth" and "keyfile" config settings on the existing servers?

Long version: My current production environment consists of three Windows 2012 R2 DataCenter servers running MongoDB version 2.4.9 as a service:

PRIMARY   - ServerA.example.org:27017
SECONDARY - ServerB.example.org:27017
ARBITER   - ServerC.example.org:27017

I have two MongoDB user accounts in the environment:

{ "_id" : ObjectId("XXXXXXXXXX"), "pwd" : "XXXXXXXXXX", "user" : "mongo-admin", "roles" : [  "userAdminAnyDatabase" ] }
{ "_id" : ObjectId("XXXXXXXXXX"), "pwd" : "XXXXXXXXXX", "user" : "mongo-cluster-admin", "roles" : [  "clusterAdmin" ] }

And the config file on the existing servers looks like this:

logpath=C:\mongo\logs\mongo.log
dbpath=C:\mongo\data
replSet=rs
oplogSize=700
port=27017
auth=true
keyFile=C:\mongo\config\keyfile.txt

I want to add ServerD and ServerE as new SECONDARY servers.

I've tried the following from my workstation and from the local console on ServerA, with and without the "auth" and "keyfile" in the config on ServerD, but I get the same result in all cases:

D:\MongoDB\bin>mongo ServerA.example.org:27017
MongoDB shell version: 2.4.7
connecting to: ServerA.example.org:27017/test
> use admin
switched to db admin
> db.auth("mongo-cluster-admin", "XXXXXXXXXX")
1
rs:PRIMARY> rs.add("ServerD.example.org:27017")
Mon May 19 11:41:42.050 count failed: { "ok" : 0, "errmsg" : "unauthorized" } at src/mongo/shell/query.js:180

The only way I've been able to get this working in a test environment is if I disable the "auth" and "keyfile" settings on all five servers, then add the new servers to the replica set using "rs.add()" on the PRIMARY, and finally re-enable "auth" and "keyfile" on all servers. This requires a couple of restarts of the MongoDB service on each box to update the config settings, which I really want to avoid.

Is there a way to add the new SECONDARY servers to the replica set without changing the "auth" and "keyfile" settings on the existing servers first?

Improve this question

1 Answer 1

Reset to default
3

Just add the new secondary with auth enabled and using identical keyfiles from the start. If you do that, they will connect, use the keyfile to authenticate, and then sync from scratch - there is no reason to change the original members of the set, just make sure that the new ones you add have the correct (identical) keyfile as the original members.

The other thing to make sure of is that the user you are authenticated as has the correct permissions to do the additions (since turning off auth would solve that issue also). For reference, the built in role with the relevant permissions is clusterManager for 2.6+ - clusterAdmin should also work, and has the advantage of existing in 2.4.

Per the comments discussion, the caveat with clusterAdmin in 2.4 is that although you have permission to run the relevant commands, you will need to have read privileges on the local database (per the Combined Access section of the 2.4 docs) to use the rs.conf() command. It's not strictly required but without it you will have to construct the entire config document from scratch for a reconfiguration command which is certainly inconvenient.

Improve this answer
8
  • Hi Adam. I tried that initially, but I got the same "unauthorized" problem. I've just had a breakthrough though by giving my "mongo-cluster-admin" account "read" access to the "local" database as per docs.mongodb.org/v2.4/reference/user-privileges/…. (That might be specific to the version I'm using - i.e. 2.4.9). I'll add an answer with details once I've finished testing it.
    – mclayton
    Commented May 20, 2014 at 13:34
  • The set members do not use that account to replicate though, so I don't see why that would make a difference (they use the internal __system user/role). The procedure I described is one I have used successfully several times, and the usual reason for an unauthorized failure is that the keyfiles are not in fact identical across the set (hence the inter-set auth fails).
    – Adam C
    Commented May 20, 2014 at 13:58
  • I think the problem was that the "mongo-cluster-admin" account I'd created didn't have the required permissions to make changes to the replica set config, as opposed to it being a problem with the replication mechanism itself. After I gave it "local: read" privilege, the account was able to modify the replica set config and it all worked perfectly.
    – mclayton
    Commented May 20, 2014 at 17:22
  • Ah, I see, you needed to have the permissions from the built-in role clusterManager to make the changes, and turning off auth as you described would remove the problem by allowing you to make any changes you require (no role restriction in place, because no roles). I added the clusterManager note to the answer to prevent confusion for others (that don't get this far in the comments).
    – Adam C
    Commented May 21, 2014 at 9:26
  • There's a little bit more to it than that... The massive gotcha is that you need the "read" role in the "local" database as well as the "clusterAdmin" role in the "admin" database. The "Combined Access" section of my previous link gives a tiny fragment of a clue about this where it says that "rs.conf()" requires read access on the local database. It doesn't say, but "rs.initiate()", "rs.add()" and "rs.addArb()" also need that role otherwise you hit the "unauthorized" error when auth is enabled. (For completeness, this is all on version 2.4.9. Things may be different for newer versions).
    – mclayton
    Commented May 22, 2014 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.