A few hours ago, we had an outage in a server which was running a mongos instance. After resolving the underlying issue, the mongos has been unable to boot with the following error:
"ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Invariant failure","attr":{"expr":"!groupAndId.groupData","file":"src/mongo/s/sharding_task_executor_pool_controller.cpp","line":134}
"ctx":"ReplicaSetMonitor-TaskExecutor","msg":"\n\n***aborting after invariant() failure\n\n"
"ctx":"ReplicaSetMonitor-TaskExecutor","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}
And now, other mongos and mongod instances that required restarts are not able to boot up as well with the same error - so in time they might all fail one by one and not be able to boot again.
Our MongoDB cluster contains 5 data shards and 1 config shard, each consisting of a 3 member replica set. Let's label each data shard A,B,C,D,E, and each instance A1,A2,A3,B1,B2,etc...
Upon closer inspection of the failing mongos instance, I noticed some odd things in the RSM Topology Change logs. The hosts seems to be mixed up between the replset. Notice the topology change for replset-B, containing 1 host from its set, but the hosts for other sets, but at the same time, the "setName" property showing replset-A.
"ctx":"ReplicaSetMonitor-TaskExecutor",
"msg":"RSM Topology Change",
"attr":{
"replicaSet":"replset-B",
"newTopologyDescription":"{
id: \"bfcaaf31-9ead-497b-85b9-1e98dbf97805\",
topologyType: \"ReplicaSetNoPrimary\",
servers: {
server-B1:27017: {
address: \"server-B1:27017\", type: \"Unknown\",
minWireVersion: 0, maxWireVersion: 0,
lastUpdateTime: new Date(-9223372036854775808),
hosts: {}, arbiters: {}, passives: {}
},
server-A1:27017: {
address: \"server-A1:27017\", type: \"Unknown\",
minWireVersion: 0, maxWireVersion: 0,
lastUpdateTime: new Date(-9223372036854775808),
hosts: {}, arbiters: {}, passives: {}
},
server-A2:27017: {
address: \"server-A2:27017\", type: \"Unknown\",
minWireVersion: 0, maxWireVersion: 0,
lastUpdateTime: new Date(-9223372036854775808),
hosts: {}, arbiters: {}, passives: {}
},
server-A3:27017: {
address: \"server-A3:27017\", type: \"Unknown\",
minWireVersion: 0, maxWireVersion: 0,
lastUpdateTime: new Date(-9223372036854775808),
hosts: {}, arbiters: {}, passives: {}
}
},
logicalSessionTimeoutMinutes: 30,
setName: \"replset-A\",
compatible: true
}",
"previousTopologyDescription":"{
id: \"2e9451c0-f13e-4d77-b9f9-6900ee5754ab\",
topologyType: \"Unknown\",
servers: {
server-B2:27017: {
address: \"server-B2:27017\", type: \"Unknown\",
minWireVersion: 0, maxWireVersion: 0,
lastUpdateTime: new Date(-9223372036854775808),
hosts: {}, arbiters: {}, passives: {}
},
server-B3:27017: {
address: \"server-B3:37017\", type: \"Unknown\",
minWireVersion: 0, maxWireVersion: 0,
lastUpdateTime: new Date(-9223372036854775808),
hosts: {}, arbiters: {}, passives: {}
}
},
compatible: true
}"
}
I ran sh.status() on a functioning mongos instance, and lo and behold, the shard-B was appearing as replset-B/server-A1:27017,server-A2:27017,server-A3:27017. I have no idea how this happened. We've had this cluster running for a couple of years now, and the last change to the topology was adding shard-E a couple of weeks ago. All the mongod.conf have the correct replSet names in them, doing rs.conf() and rs.status() on the primaries of each replica set shows no inconsistencies.
I've even tried updating the config.shards collection directly and $setting the correct hosts for the problematic shard's, both through mongos and through the config primary, but not only did it not work (the other mongos instances kept failing with the same error), but it reverted back to the old (wrong) connection string after a while (which I suspect has to do with the ShardRegistry mechanism that updates the config.shards as well).
Running db.runCommand({getShardMap: 1}) gives me the following result (which I suspect is another symptom of some conflict somewhere):
"connStrings" : {
...all good until...
"replset-B/server-A1:27017,server-A2:27017,server-A3:27017" : "shard-B",
}
I'm nearing my wit's end. Downtime is not really an option, and I was wondering if someone more knowledgeable in the inner workings of MongoDB had some insight to share on this.
