I have a 15 shard mongodb PSS cluster with huge traffic approx 1TB per shard data
I have seen that the queries get piled up on some shards which I find using db.currentOp on the mongos server.
I do this
db.currentOp().inprog.forEach( function(op) { if(op.secs_running > 10) print("opid: "+op.opid+" Operation->" + op.op + " DB->" + op.ns + " Time in Secs->" + op.secs_running); } );
A majority of the queries are any one shard ( randomly ) I login to the primary of that shard and fire rs.stepDown() all queries get released
Why ?
What should I check ?
Also I have other clusters in the same DC that do not exhibit this behaviour with almost same traffic
I use mongodb version 4.0.18 on this cluster