I have a collection which enabled sharding on 2 replica sets, rs01 and rs02. I added a new replica set rs03, it started to migrating chunks to rs03 slowly(it only migrated about 300 chunks in 4 days...)

It is weird that I do a find().limit(1) or findOne() on the collection on mongos, it hung there without response.(I need ctrl-c to escape the mongo shell on mongos).

How to check anything wrong that makes the query stuck? Because the mongo.log on the rs01 is filled with migration actions that I cannot find useful message that make the findOne() query stuck.

Moreover, I tried findOne() on the sharded collection on rs01, rs02, rs03 ,all can return document. Only hung on mongos.

Below is message from mongos by db.locks.find( { _id : "balancer" } ).pretty()

mongos> db.locks.find( { _id : "balancer" } ).pretty()
    "_id" : "balancer",
    "state" : 2,
    "who" : "mongos30-02:27017:1459106195:1804289383:Balancer:846930886",
    "ts" : ObjectId("5784b2b42e0cc50683dc622e"),
    "process" : "mongos30-02:27017:1459106195:1804289383",
    "when" : ISODate("2016-07-12T09:04:52.713Z"),
    "why" : "doing balance round"

Update: I tried adding a balancer schedule at midnight. the balancer stopped now. Even the migration thread stopped, It still failed to findOne() on mongos. but worked on each primary node.

I got log from mongos when I issue findOne() command as below:

2016-07-13T14:15:06.118+0800 I SHARDING [LockPinger] cluster mongocfg30-01:27019,mongocfg30-02:27019,mongocfg30-03:27019 pinged successfully at Wed Jul 13 14:15:05 2016 by distributed lock pinger 'mongocfg30-01:27019,mongocfg30-02:27019,mongocfg30-03:27019/mongos30-01:27017:1459049395:1804289383', sleeping for 30000ms

The cluster map is as below(some mongo version are not the same) All node OS is ubuntu 14.04 kvm, where all mongod have 20G RAM.

enter image description here enter image description here enter image description here

  • by the way, if I add filter _id to a specific id object on find(), mongos can return the record.
    – CSJ
    Commented Jul 12, 2016 at 3:57
  • Also, I tried findOne() on rs01, rs02, rs03 all can return document. Only hung on mongos
    – CSJ
    Commented Jul 12, 2016 at 5:47
  • In general migration is a background activity that does not block queries so there must be other factors in play if you can query from all shard replica sets but not via mongos. Your earlier question on chunk migration (dba.stackexchange.com/q/143265/9345) suggests you've added a new shard rs03 and there may be a significant amount of migrations happening (perhaps overwhelming your server resources). Can you provide some more details: what specific version of MongoDB server and O/S are you using (and if MongoDB 3.0.x+, what storage engine and file system)?
    – Stennie
    Commented Jul 12, 2016 at 13:36
  • Were you using the same read preference when querying via mongos vs the shards directly (i.e. reading from primary)? Are all components of your cluster (mongos and mongod) using identical MongoDB versions? Also, how many mongos do you have? Have you checked the mongos logs for any further information that might correlate with your query blocking?
    – Stennie
    Commented Jul 12, 2016 at 13:38
  • Hi, @Stennie, Yes, I use db.mycollect.findOne() on primaries and mongos.I've updated more detailed arch info. I need to check the mongos log further, Although I tail -f /var/log/mongo/mongos.log | grep mycollect when I use db.mycollect.findOne(), I got no info.
    – CSJ
    Commented Jul 13, 2016 at 2:19


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.