0

I tried to set up a shard to test this functionality and maybe migrate some of our data into it.

I followed the documentation guide https://docs.mongodb.com/v3.4/tutorial/convert-replica-set-to-replicated-shard-cluster/

So I have a first replica rs0 (containing 2 servers), a replica rs1 (containing 2 servers) and a config replica (containing 2 servers for my tests, I understand I should use at least 3 in production)

Now, when I'm connecting on mongos instance, I see that rs0 is my primary shard. And the collection I sharded contains 1,000,000 documents. rs1 contains 1 chunk which contains approximately 500,000 documents. If I execute db.partitioned_collection.count() in mongos it returns 1,500,000. Whereas I would expect it to return 1,000,000.

I get that the primary shard contains all the unsharded documents, but mongos should only query data inside rs0 chunks no ?

What am I doing wrong ?

My sh.status() result is :

--- Sharding Status ---
sharding version: {
    "_id" : 1,
    "minCompatibleVersion" : 5,
    "currentVersion" : 6,
    "clusterId" : ObjectId("5b992a952e6c702f9dd5b675")
}
shards:
    {  "_id" : "rs0",  "host" : "rs0/mongo-test:27017,mongo-test:27018",  "state" : 1 }
    {  "_id" : "rs1",  "host" : "rs1/mongo-test:27019,mongo-test:27020",  "state" : 1 }
active mongoses:
    "3.6.7" : 1
autosplit:
    Currently enabled: yes
balancer:
    Currently enabled:  yes
    Currently running:  no
    Failed balancer rounds in last 5 attempts:  0
    Migration Results for the last 24 hours:
            2 : Success
databases:
    {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
            config.system.sessions
                    shard key: { "_id" : 1 }
                    unique: false
                    balancing: true
                    chunks:
                            rs0     1
                    { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : rs0 Timestamp(1, 0)
    {  "_id" : "test",  "primary" : "rs0",  "partitioned" : true }
            test.partitioned_collection
                    shard key: { "number" : 1 }
                    unique: false
                    balancing: true
                    chunks:
                            rs0     2
                            rs1     1
                    { "number" : { "$minKey" : 1 } } -->> { "number" : 4794 } on : rs1 Timestamp(2, 0)
                    { "number" : 4794 } -->> { "number" : 9586 } on : rs0 Timestamp(2, 1)
                    { "number" : 9586 } -->> { "number" : { "$maxKey" : 1 } } on : rs0 Timestamp(1, 2)

UPDATE #1 :

It seems I completely misread the following documentation https://docs.mongodb.com/v3.6/core/sharded-cluster-shards/#primary-shard . Indeed the primary shard does not hold the unsharded documents but the unsharded collections ...

So the question now is why my primary shard (rs0) which should only contains 2 chunks, is still containing the whole collection data ?

  • The primary shard for a db only refers to ownership of unsharded collections. Since partitioned_collection is sharded, it is eligible to have data on multiple shards. mongos should be returning the count of unique documents across all shards (rs0 has 2 chunks and rs1 has 1). I notice you are referring to 3.4 docs but the question has a mongodb-3.6 tag. What is your specific MongoDB server version as reported by db.version()? Also, are you using any secondary read preferences when querying via mongos? – Stennie Sep 13 '18 at 3:31
  • I posted the wrong link, sorry, my version as returned by db.version() is 3.6.7. I'm not using any read preference, I'm querying my collection with the exact following commands : mongo --port 27030 use test db.partitioned_collection.count() – b.moyet Sep 13 '18 at 5:56
  • I think I might understand what's happening. My test (count all documents) is probably not a great one, because the query is bot using the shard key. Should every querys done to the shard collection have the shard key as a filter ? I think so, based on the query plan I analyzed, because if I know try the query db.peoples.find({number:3000}) I don't get the data in rs0 counted. – b.moyet Sep 13 '18 at 7:06
  • Well, I'm still having an issue on this, after studying query plans it seems that if the winning plan stage is SHARD_MERGE, it will count documents twice. Whereas if the stage is SINGLE_SHARD, it will give me the correct count. Thus, db.partitioned_collection.find({number:3000}) is giving me the correct count, whereas db.partitioned_collection.find({"number":{$in[4790, 4800]}}).count() is not. – b.moyet Sep 13 '18 at 8:30
0

I finally tested again to follow exactly the example at https://docs.mongodb.com/v3.6/tutorial/convert-replica-set-to-replicated-shard-cluster/ . And this time it did work... It took some time (don't know how much exactly) for rs0 to delete the moved documents, but after a few minutes (or tens of minutes) they were not there anymore.

I have tried to remember what I did differently on the first time. I'm wondering if I did insert the test data after rs0 became a part of the shard directly on the rs0 replica set. And from what I understand we must not insert/update data on shard replica sets directly.

  • Inserting documents directly into the replica set backing a sharded collection is definitely one way to skew the counts in MongoDB 3.6 -- this can create orphaned documents that aren't expected to be found on the shard. You could fix this by running the cleanupOrphaned command on an affected mongod primary. – Stennie Sep 13 '18 at 13:15
  • NOTE: the behaviour for counts on sharded collections with a query predicate (as per your examples) has been improved in MongoDB 4.0 and newer, so orphaned documents are now ignored. See SERVER-3645 in the MongoDB issue tracker for more background. – Stennie Sep 13 '18 at 13:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.