0

I am new to mongodb and this is my second post in order to build a test cluster and gain knowledge at the same time. I have a 3 nodes cluster holding server MONGO2, MONGO3 and MONGOARB1. MONGO2 and MONGO3 are in replicaset "rs0" and works great :

rs0:PRIMARY> rs.conf()
{
        "_id" : "rs0",
        "version" : 3,
        "term" : 9,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "MONGO2:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "MONGO3:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                }
        ],

I'd like to add an arbiter to my PSA cluster as described here : https://docs.mongodb.com/manual/tutorial/add-replica-set-arbiter/

enter image description here

So following the documentation:

  • Create a specific directory => OK (C:\MONGO\arb)
  • Start the arbiter, specifying the data directory and the name of the replica set to join => OK : ./mongod.exe --port 27017 --dbpath C:\MONGO\arb --replSet rs0 --bind_ip_all

"msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":24200}}

  • Connect to the primary and add the arbiter to the replica set. => KO rs.addArb("MONGOARB1:27017");

After this last command nothing happen and the error log show followings messages:

{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":21352,   "ctx":"conn5","msg":"replSetReconfig admin command received from client","attr":{"newConfig":{"_id":"rs0","version":4,"term":9,"members":[{"_id":0,"host":"MONGO2:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"MONGO3:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"MONGOARB1:27017","arbiterOnly":true}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"61d5799d9249ea1cc550e79d"}}}}}
{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":6015313, "ctx":"conn5","msg":"Replication config state is Steady, starting reconfig"}
{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigReconfiguring","oldState":"ConfigSteady"}}
{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigReconfiguring"}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":4508702, "ctx":"conn5","msg":"Waiting for the current config to propagate to a majority of nodes"}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":51815,   "ctx":"conn5","msg":"Waiting for the last committed optime in the previous config to be committed in the current config","attr":{"configOplogCommitmentOpTime":{"ts":{"$timestamp":{"t":1642004542,"i":2}},"t":9}}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":4508701, "ctx":"conn5","msg":"The current replica set config is committed","attr":{"configVersion":3,"configTerm":9,"configWaitDurationMillis":0,"oplogWaitDurationMillis":0,"configOplogCommitmentOpTime":{"ts":{"$timestamp":{"t":1642004542,"i":2}},"t":9}}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":21352,   "ctx":"conn5","msg":"replSetReconfig admin command received from client","attr":{"newConfig":{"_id":"rs0","version":4,"term":9,"members":[{"_id":0,"host":"MONGO2:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"MONGO3:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"MONGOARB1:27017","arbiterOnly":true}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"61d5799d9249ea1cc550e79d"}}}}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":6015313, "ctx":"conn5","msg":"Replication config state is Steady, starting reconfig"}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigReconfiguring","oldState":"ConfigSteady"}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigReconfiguring"}}

And after a while the command fail with:

rs0:PRIMARY> rs.addArb("MONGOARB1:27017");
assert.soon failed: function() {
        var cfg = hostport;

        var local = db.getSiblingDB("local");
        assert(local.system.replset.count() <= 1,
               "error: local.system.replset has unexpected contents");
        var c = local.system.replset.findOne();
        assert(c, "no config object retrievable from local.system.replset");

        const attemptedVersion = c.version++;

        var max = 0;
        for (var i in c.members) {
            // Omit 'newlyAdded' field if it exists in the config.
            delete c.members[i].newlyAdded;
            if (c.members[i]._id > max)
                max = c.members[i]._id;
        }
        if (isString(hostport)) {
            cfg = {_id: max + 1, host: hostport};
            if (arb)
                cfg.arbiterOnly = true;
        } else if (arb == true) {
            throw Error(
                "Expected first parameter to be a host-and-port string of arbiter, but got " +
                tojson(hostport));
        }

        if (cfg._id == null) {
            cfg._id = max + 1;
        }
        c.members.push(cfg);

        res = self._runCmd({replSetReconfig: c});
        if (res === "") {
            // _runCmd caught an exception.
            return true;
        }
        if (res.ok) {
            return true;
        }
        if (res.code === ErrorCodes.ConfigurationInProgress ||
            res.code === ErrorCodes.CurrentConfigNotCommittedYet) {
            return false;  // keep retrying
        }
        if (res.code === ErrorCodes.NewReplicaSetConfigurationIncompatible) {
            // We will retry only if this error was due to our config version being too low.
            const cfgState = local.system.replset.findOne();
            if (cfgState.version >= attemptedVersion) {
                return false;  // keep retrying
            }
        }
        // Take no action on other errors.
        return true;
    } : {
        "ok" : 0,
        "errmsg" : "Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.",
        "code" : 103,
        "codeName" : "NewReplicaSetConfigurationIncompatible",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1642005752, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1642005752, 1)
} The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: false} as the fifth argument (you can fill unused arguments with `undefined`). Running hang analyzer from assert.soon.
Skipping runHangAnalyzer: no TestData (not running from resmoke)
Error: assert.soon failed: function() {
        var cfg = hostport;

        var local = db.getSiblingDB("local");
        assert(local.system.replset.count() <= 1,
               "error: local.system.replset has unexpected contents");
        var c = local.system.replset.findOne();
        assert(c, "no config object retrievable from local.system.replset");

        const attemptedVersion = c.version++;

        var max = 0;
        for (var i in c.members) {
            // Omit 'newlyAdded' field if it exists in the config.
            delete c.members[i].newlyAdded;
            if (c.members[i]._id > max)
                max = c.members[i]._id;
        }
        if (isString(hostport)) {
            cfg = {_id: max + 1, host: hostport};
            if (arb)
                cfg.arbiterOnly = true;
        } else if (arb == true) {
            throw Error(
                "Expected first parameter to be a host-and-port string of arbiter, but got " +
                tojson(hostport));
        }

        if (cfg._id == null) {
            cfg._id = max + 1;
        }
        c.members.push(cfg);

        res = self._runCmd({replSetReconfig: c});
        if (res === "") {
            // _runCmd caught an exception.
            return true;
        }
        if (res.ok) {
            return true;
        }
        if (res.code === ErrorCodes.ConfigurationInProgress ||
            res.code === ErrorCodes.CurrentConfigNotCommittedYet) {
            return false;  // keep retrying
        }
        if (res.code === ErrorCodes.NewReplicaSetConfigurationIncompatible) {
            // We will retry only if this error was due to our config version being too low.
            const cfgState = local.system.replset.findOne();
            if (cfgState.version >= attemptedVersion) {
                return false;  // keep retrying
            }
        }
        // Take no action on other errors.
        return true;
    } : {
        "ok" : 0,
        "errmsg" : "Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.",
        "code" : 103,
        "codeName" : "NewReplicaSetConfigurationIncompatible",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1642005752, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1642005752, 1)
} The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: false} as the fifth argument (you can fill unused arguments with `undefined`). :
doassert@src/mongo/shell/assert.js:20:14
assert.soon@src/mongo/shell/assert.js:382:17
rs.add@src/mongo/shell/utils.js:1632:5
rs.addArb@src/mongo/shell/utils.js:1704:12
@(shell):1:1
rs0:PRIMARY>

Note that if I use the command "rs.add" it works great to add the node as it was a replicaset member holding datas.

7
  • Did you try rs.add( { host: "MONGOARB1:27017", arbiterOnly: true } )? Jan 12 at 21:35
  • Is folder C:\MONGO\arb empty? Compare config files, there could be difference. Option enableMajorityReadConcern has been removed in MongoDB version 5, do you still have it set? Jan 12 at 21:39
  • What is the version of MongoDB you are working with? What is the output from the rs.status() and db.hello() methods?
    – prasad_
    Jan 13 at 4:50
  • @Wenfried rs.add( { host: "MONGOARB1:27017", arbiterOnly: true } ); returns the same behavior. C:\MONGO\arb is empty when created but filed with several files after the command ./mongod.exe --port 27017 --dbpath C:\MONGO\arb --replSet rs0 --bind_ip_all (collection.wt, mongod.lock, storage.bson etc) - I am not using the option enableMajorityReadConcern.
    – Mickael
    Jan 13 at 8:36
  • @prasad before and after adding the arbiter rs.satus() & db.hello() returns the current config, it is up and running.
    – Mickael
    Jan 13 at 8:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.