1

I am trying to test mongoDB sharding, but when following the basic steps in common tutorials, I am facing a CONNECT_ERROR issue that really doesn't make sense to me.

I use four machines for this test :

  • One for the mongos process : let's call it "GOS"
  • One for the config mongod : let's call it "GOD"
  • Two for the storage (shards), that are on the same subnet, behind the same firewall : let's call them "SH1" and "SH2"

In real world, GOS, GOD, SH1 and SH2 are replaced by the matching IP addresses.

I assume that they are properly configured together, and they all can communicate. They are all using port 49029 for legacy reasons.

Before playing with any data, on GOS,

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "version" : 3,
    "minCompatibleVersion" : 3,
    "currentVersion" : 4,
    "clusterId" : ObjectId("56446b3a8f521d8fad05f4c5")
}
  shards:
    {  "_id" : "shard0000",  "host" : "SH1:49029" }
    {  "_id" : "shard0001",  "host" : "SH2:49029" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }

On GOD,

> use config
switched to db config
> db.shards.find()
{ "_id" : "shard0000", "host" : "SH1:49029" }
{ "_id" : "shard0001", "host" : "SH2:49029" }

Now consider the following sequence of instructions on GOS:

mongos> use test
switched to db test
mongos> db.createCollection("yolo")
{ "ok" : 1 }
mongos> db.yolo.insert({"test":"ok"})
mongos> db.yolo.count()
1
mongos> use admin
switched to db admin
mongos> db.adminCommand({"enableSharding" : "test"})
{ "ok" : 1 }
mongos> use test
switched to db test
mongos> db.yolo.count()
1
mongos> db.yolo.ensureIndex({"test":1})
mongos> db.yolo.count()
1
mongos> use admin
switched to db admin
mongos> db.adminCommand({"shardCollection":"test.yolo", key:{"test":1}})
{ "collectionsharded" : "test.yolo", "ok" : 1 }
mongos> sh.status()
--- Sharding Status ---
  sharding version: {
    "_id" : 1,
    "version" : 3,
    "minCompatibleVersion" : 3,
    "currentVersion" : 4,
    "clusterId" : ObjectId("56446b3a8f521d8fad05f4c5")
}
  shards:
    {  "_id" : "shard0000",  "host" : "SH1:49029" }
    {  "_id" : "shard0001",  "host" : "SH2:49029" }
  databases:
    {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
    {  "_id" : "properties",  "partitioned" : false,  "primary" : "shard0000" }
    {  "_id" : "prorating_Ozon",  "partitioned" : true,  "primary" : "shard0000" }
    {  "_id" : "test",  "partitioned" : true,  "primary" : "shard0001" }
        test.yolo
            shard key: { "test" : 1 }
            chunks:
                shard0001   1
            { "test" : { "$minKey" : 1 } } -->> { "test" : { "$maxKey" : 1 } } on : shard0001 Timestamp(1, 0)

mongos> use test
switched to db test
mongos> db.yolo.count()
setShardVersion failed host: SH2:49029 { oldVersion: Timestamp 0|0, oldVersionEpoch: ObjectId(000000000000000000000000'), errmsg: exception: socket exception [CONNECT_ERROR] for GOD:49029", code: 11002, ok: 0.0 }

The same error for each operation I tried to perform on this collection from GOS since I enabled sharding and specified the key.

Note that after inserting the data, I can open a mongo shell on SH2 :

> use test
switched to db test
> db.yolo.find()
{ "_id" : ObjectId("564493222b3e59c2861188f4"), "test" : "ok" }

It still works after GOS goes wrong.

What doesn't make much sense to me here is that it starts going wrong just after I start playing with shard keys. I think that the first insert going well eliminates the lead of firewall issues (I've had many, before). I am not sure I really understand what the db.adminCommand({"shardCollection"[…]) implies, but I don't see how it could have any influence on connectivity between my entities. I would really appreciate if someone with more experience could give me a clue of what is going on here.


Addendum, on GOS:

mongos> db.runCommand( { connPoolStats: 1 } )
{
    "hosts" : {
        "GOD:49029::0" : {
            "available" : 1,
            "created" : 1
        },
        "GOD:49029::30" : {
            "available" : 3,
            "created" : 3
        },
        "SH1:49029::0" : {
            "available" : 1,
            "created" : 2
        },
        "SH1:49029::30" : {
            "available" : 0,
            "created" : 2
        },
        "SH2:49029::0" : {
            "available" : 1,
            "created" : 2
        },
        "SH2:49029::30" : {
            "available" : 0,
            "created" : 2
        }
    },
    "replicaSets" : {

    },
    "createdByType" : {
        "master" : 12
    },
    "totalAvailable" : 6,
    "totalCreated" : 12,
    "numDBClientConnection" : 12,
    "numAScopedConnection" : 3,
    "ok" : 1
}
  • GOD:49029 — is this the host that you have specified via the --configdb option when starting your mongos? Is this host reachable directly, by exactly this hostname and port, from SH2? Please verify this. Your shard servers must be able to connect to the config server. – Dmytro Shevchenko Nov 13 '15 at 8:55
  • Hi @DmytroShevchenko and thank you for your help. As I mentioned, in my actual situation, GOS, GOD, SH1 and SH2 are replaced by the matching IP addresses. I have properly specified --configdb with the matching IP address when starting my mongos. They are all ping-able from each other. I don't really know how to check that they can be reached directly with the given port, except by launching mongo GOD:49029 from SH2, which works. – Vongo Nov 13 '15 at 13:02
  • Have you seen this thread? groups.google.com/forum/#!topic/mongodb-user/d1-sQe2_iUA you may find something useful in that discussion. – Dmytro Shevchenko Nov 13 '15 at 13:09
  • You asked how to see that everybody is reachable from every node. Check all mongos and mongod log-files for errors. If node cannot speak with other node, it will complain in the log file... As well, it will tell when it managed to contact other nodes. – JJussi Apr 4 '17 at 15:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.