1

We are running our production cluster using version 3.2.8, Our cluster consists of four mongos , three config servers and ten shards-3-node-replica-sets , we perform both primary and secondary reads.

Very frequently we hit SERVER-26159 and our mongos crashes. According to the bug report the issue was solved on 3.2.10 and we upgraded on Oct 11.

10 minutes after the upgrade completed we start receiving errors that the mongos can't connect to the shards due to timeout. Timeouts were very frequent and reduce our processing throughput to 20%. We forced to rollback to 3.2.8 and the issue gone away, we don't receive any timeout but the SERVER-26159 returned.

Oct 11 06:40:03 [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to node1-48668:48668
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Successfully connected to node0-49179:49179
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node0-49627:49627
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node0-49630:49630 - ExceededTimeLimit: Operation timed out
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node1-49631:49631
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node0-48668:48668
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-4-0] Connecting to node0-48574:48574
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node0-49631:49631 - ExceededTimeLimit: Operation timed out
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node0-49631:49631 - ExceededTimeLimit: Operation timed out
Oct 11 06:40:03 mongos16026 mongos.16026[27341]: [NetworkInterfaceASIO-TaskExecutorPool-1-0] Failed to connect to node1-48574:48574 - ExceededTimeLimit: Operation timed out

I check MongoInc for relevant jira's but I didn't find any. Has anyone else deal with this issue? Is this a known bug in 3.2.10 and is there any workaround?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.