0

In my team, we have a MongoDB 3.6 with 4 shard cluster.
My teammate built up Ops Manager and we found out at some point the queries are slow and they are all waiting for database level lock.

    "Database": {
      "acquireCount": {
        "w": 1
      },
      "acquireWaitCount": {
        "w": 1
      },
      "timeAcquiringMicros": {
        "w": 5383162
      }
    },

Dig inside and we found out the MongoDB was moving chunk and command require "W" exclusive lock at database level.
We feel quite confusing because

  1. We use hashed _id for shard key, which should be distributed evenly and every document size are quite the same. Is there any way to find out why MongoDB trigger moveChunk? Or any statistics for me to dig inside ?
  2. List all the slow query, Ops Manager didn't record the moveChunk command as slow query. How should I change my config or anything I should try ?
  3. Based on my case, should I try to disable balancer to prevent moveChunk and observe by myself ?

[Updated]. Here is the query which is lock on database "W" lock.

{
            "host" : "....", 
            "desc" : "migrateThread", 
            "active" : true, 
            "currentOpTime" : "2020-08-31T06:34:34.194+0000", 
            "opid" : "sh1:-472303473", 
            "secs_running" : 15, 
            "microsecs_running" : 15562026, 
            "op" : "none", 
            "ns" : "collection", 
            "command" : {
            }, 
            "msg" : "step 2 of 6", 
            "numYields" : 0.0, 
            "locks" : {
            }, 
            "waitingForLock" : false, 
            "lockStats" : {
                "Global" : {
                    "acquireCount" : {
                        "r" : 4, 
                        "w" : 4
                    }
                }, 
                "Database" : {
                    "acquireCount" : {
                        "w" : 3, 
                        "W" : 1
                    }, 
                    "acquireWaitCount" : {
                        "W" : 1
                    }, 
                    "timeAcquiringMicros" : {
                        "W" : 15665
                    }
                }, 
                "Collection" : {
                    "acquireCount" : {
                        "w" : 2, 
                        "W" : 1
                    }, 
                    "acquireWaitCount" : {
                        "W" : 1
                    }, 
                    "timeAcquiringMicros" : {
                        "W" : 7247
                    }
                }
            }
        }

Thanks

2 Answers 2

1

That was a lot of questions for one question, but here goes:

Is there any way to find out why MongoDB trigger moveChunk?

The balancer will trigger a chunk move for a collection when the number of chunks on one shard exceeds the number of chunks on any other shard by the migration threshold

Or any statistics for me to dig inside ?

Each balancer run is logged in the config.actionlog collection, and each individual chunk move should have 4 or 5 entries is the config.changelog collection. Each chunk move should also be logged in mongod.log for both the sending and receiving shard.

List all the slow query

Each mongod log will show all queries that run longer than 100ms, as well as all moveChunk operations.

Should I try to disable balancer to prevent moveChunk and observe by myself?

You might consider configuring a balancer window so the balancer does not run during times that you expect high load.

Regarding the lock itself, I had thought MongoDB 3.6 removed the need for the balancer to take a 'W' lock. A chunk move does place a fair bit of load on replication, which might cause other things to back up.

Edit

After posting this, I went back to take another look at the code. The only place I see a database-level 'W' lock (aka MODE_X) being taken by the migrateThread is prefaced by this comment in the mongod source:

// Hold the DBLock in X mode across creating the collection and indexes, so that a
// concurrent dropIndex cannot run between creating the collection and indexes and fail with
// IndexNotFound, though the index will get created.
// We could take the DBLock in IX mode while checking if the collection already exists and
// then upgrade it to X mode while creating the collection and indexes, but there is no way
// to upgrade a DBLock once it's taken without releasing it, so we pre-emptively take it in
// mode X.

Reading through, that function, it appears to:

  • check if the collection exists
  • if yes, verify the UUID of the collection
  • if no, create the collection
  • if the collection is empty on the local (recipient) shard, and the donor has sent a list of indexes, create those indexes

These operation should be nearly instant, so the total time the would be held should be on the order of single-digit milliseconds, at worst.

The migrateThread did have to wait for 15ms to acquired the database 'W' lock, but that doesn't really explain why the migrateThread process was running for over 4 hours, or why that other operation was waiting on the lock for nearly 2 hours.

3
  • Really thanks for your reply! It's bad to ask too many sub question in one question but i have no idea how to split it. I still found some admin query used the database "W" lock.
    – 鄭元傑
    Commented Aug 31, 2020 at 10:12
  • Really appreciate your explanation. I thought maybe it's because we set TTL index on collection and it cause imbalance. Currently I still cannot find out which query acquire "W" lock except the migrateThread. Do you have any advice on it ?
    – 鄭元傑
    Commented Sep 1, 2020 at 2:47
  • 1
    If the system appears stuck, kill the migrate operation and see if that shakes it loose.
    – Joe
    Commented Sep 1, 2020 at 4:30
2

To see your "real" chunk distribution count between shards, use command sh.status() at mongos. As you know, result should show that every shard have more or less same amount of chunks. Yes, you can disable the balancer and if you really use hashed sharding key, end result should be same as if you keep balancer running. Check that sh.status() carefully, I think that you have there something else sharded too, something what don't use hashed sharding key. On case when movechunks happen even you have hashed shardingkey is of course situtation where you have added new shard to the cluster. That new shard is empty and system starts to fill it.

1
  • Thanks for the clue. I will dig inside !
    – 鄭元傑
    Commented Aug 31, 2020 at 10:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.