5

I am running a mongo cluster.
Chunk size set is 300 MB but for today morning it is showing me in logs that chunk size is 1024 Byte. I checked in current op there also it is showing chunks of 1024 byte. I have checked with monos and on all config server chunk size is 300 MB.

Please help me to resolve the issue as it is all of the sudden bringing my shard set up down.

Here is the log from currentOp

 {
                    "opid" : "shard0000:-1945000000",
                    "active" : true,
                    "secs_running" : 0,
                    "microsecs_running" : NumberLong(72072),
                    "op" : "query",
                    "ns" : "DB20150102.locationCount",
                    "query" : {
                            "splitVector" : "DB20150102.locationCount",
                            "keyPattern" : {
                                    "articleId" : 1,
                                    "host" : 1
                            },
                            "min" : {
                                    "articleId" : { "$minKey" : 1 },
                                    "host" : { "$minKey" : 1 }
                            },
                            "max" : {
                                    "articleId" : { "$maxKey" : 1 },
                                    "host" : { "$maxKey" : 1 }
                            },
                            "maxChunkSizeBytes" : 1024,
                            "maxSplitPoints" : 2,
                            "maxChunkObjects" : 250000
                    },
                    "client_s" : "192.168.22.106:55881",
                    "desc" : "conn237027",
                    "threadId" : "0x7c6cc55db700",
                    "connectionId" : 237027,
                    "locks" : {
                            "^ibeat20150102" : "R"
                    },
                    "waitingForLock" : true,
                    "numYields" : 14,
                    "lockStats" : {
                            "timeLockedMicros" : {
                                    "r" : NumberLong(32978),
                                    "w" : NumberLong(0)
                            },
                            "timeAcquiringMicros" : {
                                    "r" : NumberLong(48048),
                                    "w" : NumberLong(0)
                            }
                    }
            },

Entry from setting

{ "_id" : "chunksize", "value" : 250 }

Error in primary shard

2015-01-02T13:04:48.386+0530 [conn237049] warning: chunk is larger than 1024 bytes because of key { articleId: "", host: "abc.com" }
2015-01-02T13:04:48.386+0530 [conn237049] warning: chunk is larger than 1024 bytes because of key { articleId: "0", host: "xyz.com" }

I'm seeing this in my mongos log for the same collection

2015-01-02T14:53:58.983+0530 [conn58] warning: splitChunk failed - cmd: { splitChunk: "DB20150102.locationCount", keyPattern: { articleId: 1, host: 1 }, min: { articleId: MinKey, host: MinKey }, max: { articleId: MaxKey, host: MaxKey }, from: "shard0000", splitKeys: [ { articleId: "", host: "abc.com" } ], shardId: "ibeat20150102.locationCount-articleId_MinKeyhost_MinKey", configdb: "192.168.24.192:27017,192.168.24.54:27017,192.168.24.55:27017" } result: { who: { _id: "DB20150102.locationCount", state: 1, who: "ibeatdb61:27017:1420185037:1475849446:conn913:869542099", ts: ObjectId('54a660afdc99ecfb22d83c27'), process: "ibeatdb61:27017:1420185037:1475849446", when: new Date(1420189871037), why: "split-{ articleId: MinKey, host: MinKey }" }, ok: 0.0, errmsg: "the collection's metadata lock is taken" }

I have taken these steps:

  1. Stop read an write on servers
  2. Stop mongos server
  3. Restarted config servers.
  4. Restarted mongos server
  5. Restarted read and write.

Still the same issue is appearing

My Primary Shard is a replica set configuration is as follows :

  1. Primary Server : 512 GB Ram 5 TB Physical memory.
  2. Secondary Server 16 GB RAM 5 TB Physical Server.
  3. Arbiter 8 GB Ram.

My secondary shard has the same configuration in its replica set.

  • did you stop the mongos on 192.168.22.106? What version is this? Same version in every single component of the cluster? Can you show the full command that got the entry from setting - you're checking the config DB through mongos? How did you change this value? – Asya Kamsky Jan 2 '15 at 22:36
  • 1
    in any case, you're conflating a number of different problems/symptoms, some of which are benign and some of which are warnings rather than errors. you don't say how this is "bringing your shard set up down"... – Asya Kamsky Jan 2 '15 at 22:40
  • Ya chunk size was changed according to Mongo docs only. And my issue is why its showing 1 KB chunk size all of sudden as my system is running for almost 1 year now. Also 1 KB chunk size caused heavy IO load due to frequent chunk transfer which caused heavy load on system and writes were impacted in heavy manner. My all mongo instance (MongoD and MOngos are running on 2.6 version only.) – viren Jan 5 '15 at 6:22
  • I mentioned Logs in following order: 1. Current Op from MongoS. 2. Config Setting queried through MongoS. 3. Primary shard Log. 4. Logs from MongoS mentioned in Current Op. An chunk size was set almost 6 months back and system was running fine till date with that. Allof teh usdden it started showing me this 1 KB size. – viren Jan 5 '15 at 6:25
  • @Asya could you please provide any input on that as i am seeing this issue in some other collection now. This collection is distributed evenly but now another collection is stuck on one shard and all mongos server is showing teh same metadata lock error continuously. I checked teh balancer lock and it looks like some time mongos is holding lock for over an hour. – viren Jan 5 '15 at 16:16
0

How did you change the chunk size? Did you follow this guide?

To recap, one should:

  1. connect to mongos
  2. switch to config db
  3. issue change chunk size command:
    db.settings.save( { _id:"chunksize", value: <sizeInMB> } )

AFAIK,

warning: chunk is larger than 1024 bytes because of key ...

won't bring down a shard, it will only result in a unbalanced cluster.

  • Ya chunk size was changed according to Mongo docs only. And my issue is why its showing 1 KB chunk size all of sudden as my system is running for almost 1 year now. Also 1 KB chunk size caused heavy IO load due to frequent chunk transfer which caused heavy load on system and writes were impacted in heavy manner. – viren Jan 5 '15 at 6:21
  • You mean chunk size was 300MB happily for a year and all of a sudden switched back to 1MB? What log did you get for the malfunctioning server? – user2829759 Jan 5 '15 at 6:38
  • not 1 Mb it was 1 KB. There was nothing in log as mentioned i have posted those logs in the question 2015-01-02T13:04:48.386+0530 [conn237049] warning: chunk is larger than 1024 bytes because of key { articleId: "", host: "zigwheels.com" } 2015-01-02T13:04:48.386+0530 [conn237049] warning: chunk is larger than 1024 bytes because of key { articleId: "0", host: "happytrips.com" } Thsi was from primary shard – viren Jan 5 '15 at 6:46
  • Can you describe more about your primary shard? For example in what way is it "down"? Because I suspect your quoted log is rather a warning than an error, and this particular warning shouldn't bring down a shard – user2829759 Jan 5 '15 at 7:13
  • I have edited my question along with shard configuration. On thsi configuration CPU uses were shown around 2000 % and writes became very slow. I check the balancer lock in config.locks collection and there MOngos was keeping locks for more than 2 Hours some time. – viren Jan 15 '15 at 7:36
0

Please use command sh.status() to confirm how many chunks you have and in wich shard. From given logs, I can see that you (should) have only ONE chunk, so it has not been splitted ever. The error message "the collection's metadata lock is taken" means that there is "lock" in the locks DB. You can see those with

db.getSisterDB('config').locks.find()

Check "when" key, to find that "old" lock document. You can remove it and after things should be better.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.