Primary MongoD hit 1M active sessions in cache

Question

Overview

We're running a sharded MongoDB cluster, using community edition v4.4.

The primary mongod in one of the data shards ('shardX' for short) has hit active sessions in cache limit of 1M and new sessions cannot be connected. The secondary mongod in 'shardX' also have a high number of stale config issues that caused all read requests blocked, the mitigation was to hide the secondaries.

Current Situation

Right now all the secondary mongod in 'shardX' are hidden. Only the primary mongod is serving the traffic. We meet two problems.

Unhiding the secondary mongod or move chunk will cause secondary mongod cannot find mongo config servers, get stale config errors, and bring the site down. Stale config log: "ok":0,"errMsg":"epoch mismatch detected for collection-xxx","errName":"StaleConfig","errCode":13388,

Cannot find config servers log: {"t":{"$date":"2023-12-22T22:17:08.689-08:00"},"s":"I", "c":"SH_REFR", "id":24100, "ctx":"ShardServerCatalogCacheLoader-19","msg":"Error refreshing cached database entry","attr":{"db":"sweeper","durationMillis":20000,"error":"FailedToSatisfyReadPreference: Could not find host matching read preference { mode: \"nearest\" } for set main-config-server"}}

Once we unhide a secondary mongod and let it serve traffic, this secondary mongod cannot find mongo config servers, and thus throw stale config errors. All the queries (including scatter and gather queries) hit this secondary mongod will fail. As a result, the site is down.

cached active sessions in primary Mongod reach the upper limit: 1 million, and cannot accept any new session. To connect to this shard, we need to connect to it by disableImplicitSessions:

mongosh primary-host-of-shardX --quiet --tls --tlsCertificateKeyFile /etc/mongo_secrets/mongocleaner_keystore.pem --tlsCAFile /etc/mongo_secrets/root_ca.crt --authenticationMechanism 'MONGODB-X509' --authenticationDatabase '$external' --disableImplicitSessions

All the connections without disabling implicit sessions are rejected.

Further Plan and Help

Since this is a critical production system, we're very concerned if stepDown the primary or restart primary node would cause the whole data shard get offline (e.g. not able to connect config server would result in election failures, etc).

Has anyone meet such issues before? Does anyone have any comments or advice for us to isolate the issue or on the next steps? Any comments/advice are much appreciated!

Solutions We tried

The solutions we have tried:

• Add 2 new replicas to the 'shardX' shard.
• Rolling restart all the mongos.
• Restart all the config servers.
• Add 2 new mongo config servers.
• Remove the backup node for 'shardX' shard
• Movechunk
• Kill all sessions for all the replicas in the topeka shard.
• Update outdated config server address in admin.system.version on the primary instance

But none of the above seemed to work.

Supplementary Info

when looking at the mongod.log of the primary mongod, we found many warnings:

{"t":{"$date":"2023-12-27T19:07:09.352-08:00"},"s":"W",  "c":"NETWORK",  "id":23019,   "ctx":"ReplicaSetMonitor-TaskExecutor","msg":"DNS resolution while connecting to peer was slow","attr":{"peer":"hostname-of-shardY:27017","durationMillis":563392569}}

we verified on the primary node of 'shardX': DNS service should be OK -- by using nslookup and dig of 'hostname-of-shardY' and 'hostname-of-main-config-server';

AND

We can do mongosh to connect to 'shardY' and 'main-config-server'.