I have an issue with our production mongo db, could you, great mongo experts hint me on where to dig further please?

The problem was detected in the application side (java):

Caused by: com.mongodb.MongoException$Network: Read operation to server failed on database myDb
    at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:298) ~[mongo-java-driver-2.13.0.jar:?]
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:269) ~[mongo-java-driver-2.13.0.jar:?]
    at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:84) ~[mongo-java-driver-2.13.0.jar:?]
    at com.mongodb.DBCollectionImpl.find(DBCollectionImpl.java:66) ~[mongo-java-driver-2.13.0.jar:?]
    at com.mongodb.DBCursor._check(DBCursor.java:498) ~[mongo-java-driver-2.13.0.jar:?]
    at com.mongodb.DBCursor._hasNext(DBCursor.java:621) ~[mongo-java-driver-2.13.0.jar:?]
    at com.mongodb.DBCursor.hasNext(DBCursor.java:657) ~[mongo-java-driver-2.13.0.jar:?]
    at org.springframework.data.mongodb.core.MongoTemplate.executeFindMultiInternal(MongoTemplate.java:1880) ~[spring-data-mongodb-1.7.2.RELEASE.jar:?]
    at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1705) ~[spring-data-mongodb-1.7.2.RELEASE.jar:?]
    at org.springframework.data.mongodb.core.MongoTemplate.doFind(MongoTemplate.java:1688) ~[spring-data-mongodb-1.7.2.RELEASE.jar:?]
    at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:601) ~[spring-data-mongodb-1.7.2.RELEASE.jar:?]
    at org.springframework.data.mongodb.core.MongoTemplate.find(MongoTemplate.java:592) ~[spring-data-mongodb-1.7.2.RELEASE.jar:?]
    at MyDao.findByUsername(ReadOnlyOverrideableMessageDAOMongoDBImpl.java:77) ~[classes/:?]
    ... 41 more
Caused by: java.net.SocketTimeoutException: Read timed out
    at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.8.0_92]
    at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) ~[?:1.8.0_92]

We use the following configuration:

  • a number of application nodes where every node has a local mongos process
  • a sharded collection where every shard is a replica set with two data nodes and an arbiter
  • mongo 3.0.10

I checked that there is a problem with one of the shards. Compared its logs to other shard's logs and the difference is that the problem shard has a number of records like below:

2017-09-11T09:22:35.195+0000 I SHARDING [conn2705886] request split points lookup for chunk myDb.myCollection { : "053f7ced916872b020c49ba5e353f7b1" } -->> { : "054fcdc6b828a49c342ce37e4040504a" }
2017-09-11T09:22:35.365+0000 W SHARDING [conn2705886] Finding the split vector for myDb.myCollection over { username: 1 } keyCount: 176602 numSplits: 1 lookedAt: 2352 took 169ms
2017-09-11T09:22:35.365+0000 I COMMAND  [conn2705886] command admin.$cmd command: splitVector { splitVector: "myDb.myCollection", keyPattern: { username: 1 }, min: { username: "053f7ced916872b020c49ba5e353f7b1" }, max: { username: "054fcdc6b828a49c342ce37e4040504a" }, maxChunkSizeBytes: 67108864, maxSplitPoints: 0, maxChunkObjects: 250000 } ntoreturn:1 keyUpdates:0 writeConflicts:0 numYields:1398 reslen:184 locks:{ Global: { acquireCount: { r: 2798 } }, Database: { acquireCount: { r: 1399 } }, Collection: { acquireCount: { r: 1399 } } } 170ms

Googling indicates that there might be a problem with documents distribution. However, target sharding key is a string and distribution is hash-based. Mongo says that there are not too many documents on the problem shard as well (from mongos):

mongos> db.myCollection.getShardDistribution()
Shard myDbSetProd3O at myDbSetProd3O/myHost1:27028,myHost2:27028
  data : 2.14GiB docs : 12089309 chunks : 67
  estimated data per chunk : 32.73MiB
  estimated docs per chunk : 180437

Shard myDbSetProd4D at myDbSetProd4D/myHost:27017,myHost2:27017
  data : 2.05GiB docs : 11597738 chunks : 66
  estimated data per chunk : 31.88MiB
  estimated docs per chunk : 175723

Shard myDbSetProd4E at myDbSetProd4E/myHost:27018,myHost:27018
  data : 2.14GiB docs : 12087041 chunks : 67
  estimated data per chunk : 32.73MiB
  estimated docs per chunk : 180403

Note: all of that started recently when a primary data node of that replica set was restarted

Could you guys hint me on what might be wrong here and what to check further?

  • About that split. Looks like that there are 176602 same user names and it cannot split that area. You are stuck, but you can change your chunk size bigger. docs.mongodb.com/manual/tutorial/… – JJussi Sep 11 '17 at 15:26
  • What do you mean by '176602 same user names'? 'Username' is a shard key and there can't be more than one document with it. As far as I understand, there were 176602 documents within the chunk and the chunk was successfully split – denis.zhdanov Sep 12 '17 at 3:11
  • Of course, there can be more than one same username, as long as index is not unique. Those users are in the same chunk and if there are too many same usernames (same hash value), finding split point is hard or impossible. maxSplitPoints: 0 – JJussi Sep 12 '17 at 3:49
  • 1
    Sorry but why did you decide that all those document have the same sharding key and that the split was unsuccessful? – denis.zhdanov Sep 12 '17 at 8:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.