1

I have a problem with my secondary, which is constantly going down, in normal replica set (primary, secondary, arbiter).

What I saw while investigating that there is one long lasting operation in database which is connected to sync with master:

{
    "host" : "primary-ip:my-port",
    "desc" : "WT RecordStoreThread: local.oplog.rs",
    "threadId" : "140599714879232",
    "active" : true,
    "currentOpTime" : "2018-04-24T09:36:40.103+0000",
    "opid" : 9222195,
    "secs_running" : NumberLong(30849),
    "microsecs_running" : NumberLong("30849456103"),
    "op" : "none",
    "ns" : "local.oplog.rs",
    "command" : {

    },
    "numYields" : 0,
    "locks" : {

    },
    "waitingForLock" : false,
    "lockStats" : {
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(1)
            }
        },
        "Database" : {
            "acquireCount" : {
                "w" : NumberLong(1)
            }
        },
        "oplog" : {
            "acquireCount" : {
                "w" : NumberLong(1)
            }
        }
    }
}

One other thing that I saw is that some of my queries didn't have indexes and there were some operations in currentOp() that were lasting around 10 sec. One more thing is strange graphs on mms monitoring interface (left is secondary, right is primary): enter image description here

Mongodb version: 3.6.0-rc3
Platform: Ubuntu 16.04
Cloud: AWS EC2
Usage: Lots of writes and updates

My question is what do you think is causing this behaviour? Is it possible that those queries without index are affecting this, and if they do, why are they affecting secondary instance and not primary? Should I just kill this long lasting operation? Any kind of lead or advice is welcome.

3
  • ,Could you update rs.status() & rs.isMaster() from replica set report. Commented Apr 24, 2018 at 11:50
  • please post your mongodb.log from all nodes during timeframe you have issues.
    – Sybil
    Commented Apr 27, 2018 at 5:59
  • Did you find anything in the mongodb logs for the affected instance? What instance type do you use on aws for this secondary?
    – Atish
    Commented Apr 29, 2018 at 16:21

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.