0

I've had a project dumped on me and I'm trying to get up to speed withMongo. But I have an immediate firefight, in that one shard in my Mongo cluster just keeps growing and growing, adding more files daily, and I can't find the cause. Our cluster is simple, 12 sharded replica sets. "rs1" is the primary shard, and it's the one that keeps gobbling up more and more disk space while the other 11 shards haven't grown at all. Our data is simple too, we just create 3 collections per day of our 3 document types. And we keep those for 90 days. So roughly at any time we have about 270 or so active collections, and nightly we drop the oldest collections as new collections are created for upcoming days.

The balancer seems to be running fine, I can see the chunk counts getting balanced in the collections normally.

Is there some way I can STOP mongo from sending any more data to my problem shard while I get a handle on what's going on? I've already tried setting a maxSize, but that hasn't prevented it from still growing. Right now rs1 is taking up about 885 2gb disk files. (note that this is Mongo 2.6)

  shards:
      {  "_id" : "rs1",  "host" : "rs1/psc1c250:39080,psc2c250:39080",  "maxSize" : 1500000 }
    {  "_id" : "rs10",  "host" : "rs10/psc1c259:39080,psc2c259:39080" }
    {  "_id" : "rs11",  "host" : "rs11/psc1c260:39080,psc2c260:39080" }
    {  "_id" : "rs12",  "host" : "rs12/psc1c261:39080,psc2c261:39080" }
    {  "_id" : "rs2",  "host" : "rs2/psc1c251:39080,psc2c251:39080" }
    {  "_id" : "rs3",  "host" : "rs3/psc1c252:39080,psc2c252:39080" }
    {  "_id" : "rs4",  "host" : "rs4/psc1c253:39080,psc2c253:39080" }
    {  "_id" : "rs5",  "host" : "rs5/psc1c254:39080,psc2c254:39080" }
    {  "_id" : "rs6",  "host" : "rs6/psc1c255:39080,psc2c255:39080" }
    {  "_id" : "rs7",  "host" : "rs7/psc1c256:39080,psc2c256:39080" }
    {  "_id" : "rs8",  "host" : "rs8/psc1c257:39080,psc2c257:39080" }
    {  "_id" : "rs9",  "host" : "rs9/psc1c258:39080,psc2c258:39080" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "prodLN",  "partitioned" : true,  "primary" : "rs1" }





rs1:PRIMARY> db.stats(1024*1024*1024)
{
    "db" : "prodLN",
    "collections" : 319,
    "objects" : 301960471,
    "avgObjSize" : 3532.3332748146363,
    "dataSize" : 993,
    "storageSize" : 1626,
    "numExtents" : 4436,
    "indexes" : 634,
    "indexSize" : 25,
    "fileSize" : 1765,
    "nsSizeMB" : 16,
    "dataFileVersion" : {
            "major" : 4,
            "minor" : 5
    },
    "extentFreeList" : {
            "num" : 1820,
            "totalSize" : 81
    },
    "ok" : 1
}
  • +1 for bad shard key. MongoDB move documents to shards based on the shard key (default is _id). If you dont randomise the shard key, repartition will be bad. (default _id is based on timestamp) – Jonathan Muller Mar 5 '16 at 11:17
  • I have answered below, but will also be voting to move to DBA, certainly belongs there rather than SO – Adam C Mar 5 '16 at 14:24
1

If you have one database with hundreds of collections (it's not clear that is the case because you have not included an sh.status() or similar but that seems to be what you are implying), then here is the likely scenario:

  • The prodLN database has rs1 as its primary shard
  • Therefore all new collections created within that database will be created on rs1
  • If you then shard the collections (data is sharded on a collection level, not a database level) the data will be migrated from rs1 to other shards, but then you must rely on MongoDB to clean up the data post-delete (this also assumes a good shard key to distribute the data well across the shards)
  • You have not mentioned what version you are running, but there are lots of answers out there about deleted space reuse already (some pointers: 1 2 3), let's just say it's not very efficient for certain usage patterns (including yours)
  • Someone has added a maxSize parameter to the database, so it will be considered "full" and never be a target for chunk migrations, but that does not override the primary shard behavior when creating new databases, they still go on the primary shard, take up space, then get moved off by the balancer

What can you do? Well, you have a few options, but they require changes and/or downtime for the use of the current database:

  • Start using more than one database, make sure that the new databases have different primary shards (use movePrimary if not), start writing data there, maybe rotate around several
  • You can use movePrimary on the current database, but you have to stop writing to any unsharded collections (read the behavior section in the link above). Depending on how much data there is in unsharded collections this could take some time.
  • You can attempt to reclaim space on the problem shard to buy yourself some time. If you want to minimize downtime then I would recommend a rolling resync but this is only possible if you have 3 voting members in each set - you may only have two based on your output, unless arbiters have been excluded. You could add a third member temporarily to the problem shard to perform the resyncs, then remove it later.
0

Thanks so much for the ideas. To add some details, the replica sets do have an arbiter, so there are 3 members. The nature of this data cluster also means I can have downtime if needed for maintenance, since this is just an internally used instrumentation data repository. We're using v2.6 and the only content of note is in these sharded collections, so if I were to movePrimary, I wouldn't expect that to take a lot of time.

The previous engineer (who's gone) had setup some nightly maintenance which makes sure the 3 daily collections are pre-created for 14 days forward. And the nightly job also pre-splits one of those collections which gets millions of documents added daily, to spread the loading. She also has it setup so that for the other 2 collections, the 1 "seed" chunk is moved from shard to shard, so tomorrow it might be in rs5 and the next day in rs6, in an effort I believe to prevent what you were suggesting, that all inserts were starting in rs1.

This behavior of rs1 growing non-stop has only started within the past 2 months, though the data coming in hasn't changed, which is why I was concerned that something isn't working right, perhaps with deleting old documents, even though I can see that the old collections are getting dropped.

Thanks again for the advice! I'm learning as fast as I can, but worried this thing will crash into a wall before I learn enough. :-)

  • You shouldn't add updates as an answer (unless they contain an answer), similarly thanks are discouraged, "me too" posts etc. If you need to add more information to the question that is beyond the scope of comments, then you should edit the question instead. If you want to thank people, upvote their answers instead ;) – Adam C Sep 13 '16 at 16:20
0

Without knowing result of your sh.status() command, I think your problem is that you don't shard those new collections and that's why they are piling up to that "primary" shard. Always when you create new collection, you must shard it!!

You cannot say "shard this DB" and after that, mongodb would automatically shard all collections on that DB... Every collection what you want to be sharded, must be shard.

HOWEVER, you can always say "shard this collection", even that collection don't exists yet! When you insert first document, THEN that collection is created and sharded instantaneously!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy