2

I'm running a shared MongoDB 2.4 cluster. I'm currently in a process of exhausting a shard and I'm down to the last 15 chunks needs to be migrated out of 9660.

My issue is that these chunks are too big as the shard key wasn't the best one, and dumping/restoring the 2 collections that has these issues is not an option at this moment.

I've tried to manually split the chunks, but it keeps on failing with this error:
{ "cause" : { }, "ok" : 0, "errmsg" : "split failed" }

I also tried to raise the chunk size, but that didn't work out as well.
Is there another solution I might be missing?

EDIT - Added chunk information as requested - Based on AdamC function.

mongos> AllChunkInfo("db.coll1", true)
ChunkID,Shard,ChunkSize,ObjectsInChunk
db.coll1-company_name_-6727988435900812827,shard0003,148256192,248752
db.coll1-company_name_-6615369920888926287,shard0003,88895188,149153
db.coll1-company_name_-6595735839523245520,shard0003,87448696,146726
db.coll1-company_name_-6490024596613592713,shard0003,32099368,53858
db.coll1-company_name_-3125507653387230887,shard0003,89171732,149617
db.coll1-company_name_-3084465672933415339,shard0003,105551600,177100
db.coll1-company_name_-2997585070019050064,shard0003,120763904,202624
db.coll1-company_name_-2993901072357262199,shard0003,86105908,144473
db.coll1-company_name_-2543788723715016046,shard0003,123507888,207228
db.coll1-company_name_4754688535895874026,shard0003,91061648,152788
db.coll1-company_name_5062850413490905708,shard0003,118145676,198231
db.coll1-company_name_5483223634690519314,shard0003,111475244,187039
Summary Chunk Information
Total Chunks: 12
Average Chunk Size (bytes): 100206920.33333333
Empty Chunks: 0
Average Chunk Size (non-empty): 100206920.33333333

mongos> AllChunkInfo("db.coll2", true)
ChunkID,Shard,ChunkSize,ObjectsInChunk
db.coll2-company_name_-5389355967913336416,shard0003,95282040,94808
Summary Chunk Information
Total Chunks: 1
Average Chunk Size (bytes): 95282040
Empty Chunks: 0
Average Chunk Size (non-empty): 95282040

mongos> AllChunkInfo("db.coll3", true)
ChunkID,Shard,ChunkSize,ObjectsInChunk
db.coll3-company_name_-3231146661862124155,shard0003,101878644,173263
db.coll3-company_name_658930972165413978,shard0003,142545900,242425
Summary Chunk Information
Total Chunks: 2
Average Chunk Size (bytes): 122212272
Empty Chunks: 0
Average Chunk Size (non-empty): 122212272

Thanks,
Meny

  • How this chunks look like? i mean what are the bounds? – Antonios Dec 13 '14 at 12:14
  • 1
    Yes, please post the relevant portion for those remaining jumbo chunks from the output of sh.status(true). Then, the function I wrote here will give you an idea of how big they actually are and how many objects are in them, so if you could run this and post the info for the relevant jumbo chunks that will give us all we need to answer your question: dba.stackexchange.com/a/52417/6441 – Adam C Dec 13 '14 at 13:00
  • @AdamC, I ran your function, I've just changed it to get chunks from the "problematic" shard only. The results were added to the main question – Meny Issakov Dec 13 '14 at 21:10
2

I was able to complete the exhaust by increasing the chunk size and removing the jumbo tag from the problematic chunks.
After getting the results from AdamC function, I found out the biggest chunk of that shard was ~148 MB of size.
After figuring this one out, the first step is connect to the config database and change the max chunk size:

use config
db.settings.save( { _id:"chunksize", value: 150 } )

The second step was to clear the jumbo flag from the relevant chunks. This can be found in the MongoDB Documentation: Clear jumbo flag

Once the chunks were no longer marked as 'jumbo' the balancer had no problem migrating them to another shard.
When all of this was done, I restored the chunkSize back to 64

db.settings.save( { _id:"chunksize", value: 64 } )

As I do not want that all of my chunks will be as big. when possible, I'll correct the hashed shard key on the problematic collections to create a more divisible shard keys.

Thank you AdamC, your initial guidance set me on the correct path to solve this problem.

  • You should share the procedure here as an answer and then accept it, others can then upvote it if it helps them, - I asked for the information above to basically give you that procedure (turn off balancer, find the largest chunk, increase chunk size to greater than that, remove jumbo tag, restart balancer), since you figured it and saved me the trouble, you should at least get some reputation out of it :) – Adam C Dec 14 '14 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.