3

I am new to mongodb and this is my second post in order to build a test cluster and gain knowledge at the same time. I have a 3 nodes cluster holding server MONGO2, MONGO3 and MONGOARB1. MONGO2 and MONGO3 are in replicaset "rs0" and works great :

rs0:PRIMARY> rs.conf()
{
        "_id" : "rs0",
        "version" : 3,
        "term" : 9,
        "members" : [
                {
                        "_id" : 0,
                        "host" : "MONGO2:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                },
                {
                        "_id" : 1,
                        "host" : "MONGO3:27017",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        "priority" : 1,
                        "tags" : {

                        },
                        "secondaryDelaySecs" : NumberLong(0),
                        "votes" : 1
                }
        ],

I'd like to add an arbiter to my PSA cluster as described here : https://docs.mongodb.com/manual/tutorial/add-replica-set-arbiter/

enter image description here

So following the documentation:

  • Create a specific directory => OK (C:\MONGO\arb)
  • Start the arbiter, specifying the data directory and the name of the replica set to join => OK : ./mongod.exe --port 27017 --dbpath C:\MONGO\arb --replSet rs0 --bind_ip_all

"msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":24200}}

  • Connect to the primary and add the arbiter to the replica set. => KO rs.addArb("MONGOARB1:27017");

After this last command nothing happen and the error log show followings messages:

{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":21352,   "ctx":"conn5","msg":"replSetReconfig admin command received from client","attr":{"newConfig":{"_id":"rs0","version":4,"term":9,"members":[{"_id":0,"host":"MONGO2:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"MONGO3:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"MONGOARB1:27017","arbiterOnly":true}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"61d5799d9249ea1cc550e79d"}}}}}
{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":6015313, "ctx":"conn5","msg":"Replication config state is Steady, starting reconfig"}
{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigReconfiguring","oldState":"ConfigSteady"}}
{"t":{"$date":"2022-01-12T17:33:29.038+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigReconfiguring"}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":4508702, "ctx":"conn5","msg":"Waiting for the current config to propagate to a majority of nodes"}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":51815,   "ctx":"conn5","msg":"Waiting for the last committed optime in the previous config to be committed in the current config","attr":{"configOplogCommitmentOpTime":{"ts":{"$timestamp":{"t":1642004542,"i":2}},"t":9}}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":4508701, "ctx":"conn5","msg":"The current replica set config is committed","attr":{"configVersion":3,"configTerm":9,"configWaitDurationMillis":0,"oplogWaitDurationMillis":0,"configOplogCommitmentOpTime":{"ts":{"$timestamp":{"t":1642004542,"i":2}},"t":9}}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":21352,   "ctx":"conn5","msg":"replSetReconfig admin command received from client","attr":{"newConfig":{"_id":"rs0","version":4,"term":9,"members":[{"_id":0,"host":"MONGO2:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":1,"host":"MONGO3:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"tags":{},"secondaryDelaySecs":0,"votes":1},{"_id":2,"host":"MONGOARB1:27017","arbiterOnly":true}],"protocolVersion":1,"writeConcernMajorityJournalDefault":true,"settings":{"chainingAllowed":true,"heartbeatIntervalMillis":2000,"heartbeatTimeoutSecs":10,"electionTimeoutMillis":10000,"catchUpTimeoutMillis":-1,"catchUpTakeoverDelayMillis":30000,"getLastErrorModes":{},"getLastErrorDefaults":{"w":1,"wtimeout":0},"replicaSetId":{"$oid":"61d5799d9249ea1cc550e79d"}}}}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":6015313, "ctx":"conn5","msg":"Replication config state is Steady, starting reconfig"}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigReconfiguring","oldState":"ConfigSteady"}}
{"t":{"$date":"2022-01-12T17:33:29.246+01:00"},"s":"I",  "c":"REPL",     "id":6015317, "ctx":"conn5","msg":"Setting new configuration state","attr":{"newState":"ConfigSteady","oldState":"ConfigReconfiguring"}}

And after a while the command fail with:

rs0:PRIMARY> rs.addArb("MONGOARB1:27017");
assert.soon failed: function() {
        var cfg = hostport;

        var local = db.getSiblingDB("local");
        assert(local.system.replset.count() <= 1,
               "error: local.system.replset has unexpected contents");
        var c = local.system.replset.findOne();
        assert(c, "no config object retrievable from local.system.replset");

        const attemptedVersion = c.version++;

        var max = 0;
        for (var i in c.members) {
            // Omit 'newlyAdded' field if it exists in the config.
            delete c.members[i].newlyAdded;
            if (c.members[i]._id > max)
                max = c.members[i]._id;
        }
        if (isString(hostport)) {
            cfg = {_id: max + 1, host: hostport};
            if (arb)
                cfg.arbiterOnly = true;
        } else if (arb == true) {
            throw Error(
                "Expected first parameter to be a host-and-port string of arbiter, but got " +
                tojson(hostport));
        }

        if (cfg._id == null) {
            cfg._id = max + 1;
        }
        c.members.push(cfg);

        res = self._runCmd({replSetReconfig: c});
        if (res === "") {
            // _runCmd caught an exception.
            return true;
        }
        if (res.ok) {
            return true;
        }
        if (res.code === ErrorCodes.ConfigurationInProgress ||
            res.code === ErrorCodes.CurrentConfigNotCommittedYet) {
            return false;  // keep retrying
        }
        if (res.code === ErrorCodes.NewReplicaSetConfigurationIncompatible) {
            // We will retry only if this error was due to our config version being too low.
            const cfgState = local.system.replset.findOne();
            if (cfgState.version >= attemptedVersion) {
                return false;  // keep retrying
            }
        }
        // Take no action on other errors.
        return true;
    } : {
        "ok" : 0,
        "errmsg" : "Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.",
        "code" : 103,
        "codeName" : "NewReplicaSetConfigurationIncompatible",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1642005752, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1642005752, 1)
} The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: false} as the fifth argument (you can fill unused arguments with `undefined`). Running hang analyzer from assert.soon.
Skipping runHangAnalyzer: no TestData (not running from resmoke)
Error: assert.soon failed: function() {
        var cfg = hostport;

        var local = db.getSiblingDB("local");
        assert(local.system.replset.count() <= 1,
               "error: local.system.replset has unexpected contents");
        var c = local.system.replset.findOne();
        assert(c, "no config object retrievable from local.system.replset");

        const attemptedVersion = c.version++;

        var max = 0;
        for (var i in c.members) {
            // Omit 'newlyAdded' field if it exists in the config.
            delete c.members[i].newlyAdded;
            if (c.members[i]._id > max)
                max = c.members[i]._id;
        }
        if (isString(hostport)) {
            cfg = {_id: max + 1, host: hostport};
            if (arb)
                cfg.arbiterOnly = true;
        } else if (arb == true) {
            throw Error(
                "Expected first parameter to be a host-and-port string of arbiter, but got " +
                tojson(hostport));
        }

        if (cfg._id == null) {
            cfg._id = max + 1;
        }
        c.members.push(cfg);

        res = self._runCmd({replSetReconfig: c});
        if (res === "") {
            // _runCmd caught an exception.
            return true;
        }
        if (res.ok) {
            return true;
        }
        if (res.code === ErrorCodes.ConfigurationInProgress ||
            res.code === ErrorCodes.CurrentConfigNotCommittedYet) {
            return false;  // keep retrying
        }
        if (res.code === ErrorCodes.NewReplicaSetConfigurationIncompatible) {
            // We will retry only if this error was due to our config version being too low.
            const cfgState = local.system.replset.findOne();
            if (cfgState.version >= attemptedVersion) {
                return false;  // keep retrying
            }
        }
        // Take no action on other errors.
        return true;
    } : {
        "ok" : 0,
        "errmsg" : "Reconfig attempted to install a config that would change the implicit default write concern. Use the setDefaultRWConcern command to set a cluster-wide write concern and try the reconfig again.",
        "code" : 103,
        "codeName" : "NewReplicaSetConfigurationIncompatible",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1642005752, 1),
                "signature" : {
                        "hash" : BinData(0,"AAAAAAAAAAAAAAAAAAAAAAAAAAA="),
                        "keyId" : NumberLong(0)
                }
        },
        "operationTime" : Timestamp(1642005752, 1)
} The hang analyzer is automatically called in assert.soon functions. If you are *expecting* assert.soon to possibly fail, call assert.soon with {runHangAnalyzer: false} as the fifth argument (you can fill unused arguments with `undefined`). :
doassert@src/mongo/shell/assert.js:20:14
assert.soon@src/mongo/shell/assert.js:382:17
rs.add@src/mongo/shell/utils.js:1632:5
rs.addArb@src/mongo/shell/utils.js:1704:12
@(shell):1:1
rs0:PRIMARY>

Note that if I use the command "rs.add" it works great to add the node as it was a replicaset member holding datas.

Improve this question
8
  • Did you try rs.add( { host: "MONGOARB1:27017", arbiterOnly: true } )?
    – Wernfried Domscheit
    Commented Jan 12, 2022 at 21:35
  • Is folder C:\MONGO\arb empty? Compare config files, there could be difference. Option enableMajorityReadConcern has been removed in MongoDB version 5, do you still have it set?
    – Wernfried Domscheit
    Commented Jan 12, 2022 at 21:39
  • What is the version of MongoDB you are working with? What is the output from the rs.status() and db.hello() methods?
    – prasad_
    Commented Jan 13, 2022 at 4:50
  • @Wenfried rs.add( { host: "MONGOARB1:27017", arbiterOnly: true } ); returns the same behavior. C:\MONGO\arb is empty when created but filed with several files after the command ./mongod.exe --port 27017 --dbpath C:\MONGO\arb --replSet rs0 --bind_ip_all (collection.wt, mongod.lock, storage.bson etc) - I am not using the option enableMajorityReadConcern.
    – Mickael
    Commented Jan 13, 2022 at 8:36
  • @prasad before and after adding the arbiter rs.satus() & db.hello() returns the current config, it is up and running.
    – Mickael
    Commented Jan 13, 2022 at 8:51

3 Answers 3

Reset to default
3

In your primary mongodb shell, execute following:

db.adminCommand({
   "setDefaultRWConcern" : 1,
   "defaultWriteConcern" : {
     "w" : 1
   }
 })

this should resolve the issue.

Improve this answer
1
  • 1
    The above command solves the issue. It would be better if you added an explanation to your answer of what exactly the above command does.
    – Vaibhavraj Roham
    Commented Jun 1, 2022 at 20:18
2

As Mohit mentioned above, quick answer could be

db.adminCommand({
   "setDefaultRWConcern" : 1,
   "defaultWriteConcern" : {
     "w" : 1
   }
 })

But make sure you understand what WriteConcern is. In short to implement TransactionalModel Mongo enforce replica/shards members to ackowledge/confirm Write operation.

In replicaset w could be

  • 0 if no acknowledgment from any members of replicaset required
  • 1 if standalone/primary node only need to acknowledge write operation.
  • "majority" if majority should acknowledge. (its 2 f.e. in 3 members ReplicaSet)
  • X if required...
Improve this answer
0

Had the same problem on existing cluster with mongo 5.0.12. I intentionally installed mongo 6.0 on arbiter to upgrade all nodes to 6.0. There was no errors, except for

"msg":"Failed to refresh key cache","attr":{"error":"NotYetInitialized: Cannot use non-local read concern until replica set is finished initializing.","nextWakeupMillis":24200}}

So, one should join member with matching version and upgrade secondaries only after sync.

Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.