2

I have problem on my Sharding. There are a couple of long running queries observed, running for more than 3 hours. My question is, are the below things interrelated? What is the issue? How do I fix this issue?

MongoDB Version: 3.0.8
WiredTiger Engine.

Long running Queries are:

{
    "desc" : "migrateThread",
    "threadId" : "0x95d26a000",
    "opid" : "u001b_s2_rs:-1345098410",
    "active" : true,
    "secs_running" : 9360,
    "microsecs_running" : NumberLong("9360370895"),
    "op" : "none",
    "ns" : "deviceStateEvent.deviceStateEvent",
    "query" : {

    },
    "msg" : "step 2 of 5",
    "numYields" : 0,
    "locks" : {

    },
    "waitingForLock" : false,
    "lockStats" : {
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(126582),
                "w" : NumberLong(126580)
            }
        },
        "Database" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(126579),
                "W" : NumberLong(1)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(63290),
                "W" : NumberLong(1)
            }
        },
        "oplog" : {
            "acquireCount" : {
                "w" : NumberLong(63288)
            }
        }
    }
}

{
    "desc" : "conn514744",
    "threadId" : "0x4423e3400",
    "connectionId" : 514744,
    "opid" : "u001b_s0_rs:2109828171",
    "active" : true,
    "secs_running" : 9360,
    "microsecs_running" : NumberLong("9360954810"),
    "op" : "query",
    "ns" : "deviceStateEvent.deviceStateEvent",
    "query" : {
        "$msg" : "query not recording (too large)"
    },
    "client_s" : "10.99.8.121:45784",
    "msg" : "step 3 of 6",
    "numYields" : 1063,
    "locks" : {

    },
    "waitingForLock" : false,
    "lockStats" : {
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(2130),
                "w" : NumberLong(1),
                "R" : NumberLong(1)
            }
        },
        "Database" : {
            "acquireCount" : {
                "r" : NumberLong(1064),
                "w" : NumberLong(1)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "r" : NumberLong(1064),
                "W" : NumberLong(1)
            }
        }
    }    

Below is from the mongos.log:

-----  END BACKTRACE  -----
2016-06-21T21:13:10.091-0400 I NETWORK  [conn10484] call failed to: abcdffghh.i.xxxxxx.com:10008 exception
2016-06-21T21:13:10.294-0400 I -        [conn10484] Assertion failure p src/mongo/client/dbclientinterface.h 1443
2016-06-21T21:13:10.297-0400 I CONTROL  [conn10484]
 0xa5f342 0xa03b29 0x9e820a 0x62479b 0x669edb 0x636200 0x64113b 0x644871 0x980b86 0x96bb22 0x5ca14a 0xa0fe09 0x7fa3fffb9182 0x7fa3fea8047d


 2016-06-21T21:14:47.480-0400 I ACCESS   [conn1] Successfully authenticated as principal mmsagent on admin
2016-06-21T21:15:01.521-0400 I NETWORK  [mongosMain] connection accepted from 10.99.0.72:36839 #10487 (4 connections now open)
2016-06-21T21:15:01.539-0400 I ACCESS   [conn10487] Successfully authenticated as principal procon on admin
2016-06-21T21:15:01.547-0400 I NETWORK  [conn10487] end connection 10.99.0.72:36839 (3 connections now open)
2016-06-21T21:15:01.867-0400 I NETWORK  [mongosMain] connection accepted from 10.99.0.212:59379 #10488 (4 connections now open)
2016-06-21T21:15:01.886-0400 I ACCESS   [conn10488] Successfully authenticated as principal procon on admin
2016-06-21T21:15:07.113-0400 I SHARDING [LockPinger] cluster abcdffghh.i.xxxxxx.com:10008,abcdffghh.i.xxxxxx.com:10008,abcdffghh.i.xxxxxx.com:10008 pinged success

Balancer Message

Currently enabled:  no
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                116 : Success
                1 : Failed with error 'chunk too big to move', from u001b_s4_rs to u001b_s3_rs
                9 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 7 deletes from previous migration', from u001b_s2_rs to u001b_s4_rs
                402 : Failed with error 'data transfer error', from u001b_s0_rs to u001b_s3_rs
                322 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 6 deletes from previous migration', from u001b_s2_rs to u001b_s4_rs
                8 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 7 deletes from previous migration', from u001b_s0_rs to u001b_s4_rs
                6 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 8 deletes from previous migration', from u001b_s0_rs to u001b_s4_rs
                1 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 8 deletes from previous migration', from u001b_s1_rs to u001b_s4_rs
                1 : Failed with error 'chunk too big to move', from u001b_s4_rs to u001b_s1_rs
                2 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 7 deletes from previous migration', from u001b_s1_rs to u001b_s4_rs
                387 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 6 deletes from previous migration', from u001b_s0_rs to u001b_s4_rs
                35 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 2 deletes from previous migration', from u001b_s0_rs to u001b_s2_rs
                15 : Failed with error 'chunk too big to move', from u001b_s0_rs to u001b_s3_rs
                2 : Failed with error 'chunk too big to move', from u001b_s0_rs to u001b_s2_rs
                192 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 6 deletes from previous migration', from u001b_s1_rs to u001b_s4_rs
                23 : Failed with error 'chunk too big to move', from u001b_s0_rs to u001b_s1_rs
                5 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 8 deletes from previous migration', from u001b_s2_rs to u001b_s4_rs
                77 : Failed with error 'moveChunk failed to engage TO-shard in the data transfer: can't accept new chunks because  there are still 1 deletes from previous migration', from u001b_s0_rs to u001b_s2_rs
1
+200

I'm not sure if you're using cursors on your client side but due to your long query, you could be hitting a internal timeout of 10 minutes. If you do check out Jira Ticket SERVER-3090 on Mongo.

If you are not using cursors then you can look at changing the _waitForDelete option in the balancers so the primary waits for the replicas to process the delete request from the opslog before it takes in the next operation, kind of like synchronous mirroring or availability groups in SQL Server.

NOTE THAT THIS WILL CAUSE THE PRIMARY SERVER TO WAIT FOR THE COMMANDS TO COMPLETE ON THE REPLICAS BEFORE TAKING IN NEW DATA. You might not care since you already wait hours to run queries, but if you're running some kind of prod system that needs to be fast this is not a good option.

Otherwise you're left with maximizing your replicas performance or literally stepping down a primary to clear the delete log. The newly promoted primary will not have those delete commands so it won't be waiting anymore and a election is very fast in MongoDB. Be sure to clean up orphan records afterwards. Honestly it looks like your primary and replica's can't keep up with each other currently though. Is your wiretiger environment optimized?

How to manage Mongo Balancer: https://docs.mongodb.com/manual/tutorial/manage-sharded-cluster-balancer/

example:

db.settings.update( { _id: "balancer" }, { $set : { "_waitForDelete":true } } ,  { upsert: true } )
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.