1

I'm setting up an instance of MongoDB with 3 shards. I've setup everything seemingly correctly by following the docs and comparing my config with that of a friend who has a similar setup but with a lot more shards.

The problem is that as soon as I shard a collection, db.collection.findOne() stops working and returns

2016-02-26T10:54:45.865+0100 E QUERY    [thread1] Error: error: {
        "ok" : 0,
        "errmsg" : "None of the hosts for replica set rs0 could be contacted.",
        "code" : 71
} :
_getErrorWithCode@src/mongo/shell/utils.js:23:13
DBCommandCursor@src/mongo/shell/query.js:679:1
DBQuery.prototype._exec@src/mongo/shell/query.js:105:28
DBQuery.prototype.hasNext@src/mongo/shell/query.js:267:5
DBCollection.prototype.findOne@src/mongo/shell/collection.js:215:12
@(shell):1:1

Here's the process I followed.

use testdatabase
sh.enableSharding("testdatabase")

This step returns

{ "ok" : 1 }

Then I do

sh.shardCollection("testdatabase.testcollection", {"testId": "hashed"})

Which returns

{ "collectionsharded" : "testdatabase.testcollection", "ok" : 1 }

At this point I do

sh.status()

on the config server, which lists all the shards and number of active mongoses correctly. This is its output:

--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("56a242fbc6293301b0ad4dfd")
}
  shards:
        {  "_id" : "rs1",  "host" : "rs1/pln126.example.com:27017,pln127.example.com:27017" }
        {  "_id" : "rs2",  "host" : "rs2/pln128.example.com:27017,pln129.example.com:27017" }
        {  "_id" : "rs3",  "host" : "rs3/pln130.example.com:27017,pln131.example.com:27017" }
  active mongoses:
        "3.2.1" : 3
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  5
        Last reported error:  Need to swap sharding catalog manager.  Config server reports that it is in replica set mode, but we are still using the legacy SCCC protocol for config server communication
        Time of Reported error:  Fri Feb 26 2016 15:58:07 GMT+0100 (CET)
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "testdatabase",  "primary" : "rs1",  "partitioned" : true }
                testdatabase.testcollection
                        shard key: { "testid" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                rs1     3
                        { "testid" : { "$minKey" : 1 } } -->> { "testid" : NumberLong("-3074457345618258602") } on : rs1 Timestamp(1, 0)
                        { "testid" : NumberLong("-3074457345618258602") } -->> { "testid" : NumberLong("3074457345618258602") } on : rs1 Timestamp(1, 1)
                        { "testid" : NumberLong("3074457345618258602") } -->> { "testid" : { "$maxKey" : 1 } } on : rs1 Timestamp(1, 2)

Please, note that the "Need to swap sharding catalog manager." error goes away if I change the format of mongos config file. Currently, it's in the following format - https://docs.mongodb.org/manual/reference/configuration-options/ But that error disappears if I change it to a different format (couldn't find it online), but the change would be something like this

security:
  clusterAuthMode: keyFile
  keyFile: /etc/mongod.key

would change to

keyFile=/etc/mongod.key

and I would not get that error anymore, even if the mongod configs for the other servers are still in the first format as described in the mongodb docs.

But when changed to a different one. E.g. from So basically it's placing all the chunks in the primary and won't let me do any queries on the collection. However, if I try to login to mongo on one of the config servers or one of the shards, I connect successfully.

Any ideas? Or do you need more info or something?

Thanks.

EDIT:

As requested, here's the result of rs.status()

{
        "set" : "rs0",
        "date" : ISODate("2016-02-26T14:51:59.977Z"),
        "myState" : 1,
        "term" : NumberLong(7),
        "configsvr" : true,
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "pln135.example.com:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 20162,
                        "optime" : {
                                "ts" : Timestamp(1456498319, 9),
                                "t" : NumberLong(7)
                        },
                        "optimeDate" : ISODate("2016-02-26T14:51:59Z"),
                        "electionTime" : Timestamp(1456478170, 1),
                        "electionDate" : ISODate("2016-02-26T09:16:10Z"),
                        "configVersion" : 1,
                        "self" : true
                },
                {
                        "_id" : 1,
                        "name" : "pln136.example.com:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 20147,
                        "optime" : {
                                "ts" : Timestamp(1456498312, 1),
                                "t" : NumberLong(7)
                        },
                        "optimeDate" : ISODate("2016-02-26T14:51:52Z"),
                        "lastHeartbeat" : ISODate("2016-02-26T14:51:58.352Z"),
                        "lastHeartbeatRecv" : ISODate("2016-02-26T14:51:58.351Z"),
                        "pingMs" : NumberLong(1),
                        "syncingTo" : "pln137.example.com:27017",
                        "configVersion" : 1
                },
                {
                        "_id" : 2,
                        "name" : "pln137.example.com:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 20160,
                        "optime" : {
                                "ts" : Timestamp(1456498312, 1),
                                "t" : NumberLong(7)
                        },
                        "optimeDate" : ISODate("2016-02-26T14:51:52Z"),
                        "lastHeartbeat" : ISODate("2016-02-26T14:51:58.683Z"),
                        "lastHeartbeatRecv" : ISODate("2016-02-26T14:51:58.682Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "pln135.example.com:27017",
                        "configVersion" : 1
                }
        ],
        "ok" : 1
}

Please, bear in mind, I've configured the port to be that one, it's not a mistake. I've also replaced my domain with "example" for privacy concerns.

Current mongos.conf on the App Servers.

sharding:
  configDB: "pln135.example.com:27017,pln136.example.com:27017,pln137.example.com:27017"
security:
  clusterAuthMode: keyFile
  keyFile: /etc/mongod.key

Alternative mongos.conf I've tried which doesn't give the "Need to swap sharding catalog manager."

configdb=rs0/pln135.example.com:27017,pln136.example.com:27017,pln137.example.com:27017
keyFile=/etc/mongod.key

Here's a mongod.conf from the config servers

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

#processManagement:

security:
  authorization: enabled
  keyFile: /etc/mongod.key
  clusterAuthMode: keyFile

#operationProfiling:

replication:
  replSetName: rs0

sharding:
  clusterRole: configsvr

## Enterprise-Only Options:

#auditLog:

#snmp:

Here's a mongod.conf from a shard.

# mongod.conf

# for documentation of all options, see:
#   http://docs.mongodb.org/manual/reference/configuration-options/

# Where and how to store data.
storage:
  dbPath: /var/lib/mongodb
  journal:
    enabled: true
#  engine:
#  mmapv1:
#  wiredTiger:

# where to write logging data.
systemLog:
  destination: file
  logAppend: true
  path: /var/log/mongodb/mongod.log

# network interfaces
net:
  port: 27017
  bindIp: 0.0.0.0

#processManagement:

security:
  authorization: enabled
  keyFile: /etc/mongod.key

#operationProfiling:

replication:
  replSetName: rs1

sharding:
  clusterRole: shardsvr

## Enterprise-Only Options:

#auditLog:

#snmp:
  • "None of the hosts for replica set rs0 could be contacted." is more than just a subtle hint what might be wrong. You want to check that each node in the cluster can resolve the name you used in the replica set configuration for the members. Furthermore, you need to make the according ports accessible. Please add the output of sh.status() and rs.status() for rs0 to your question. – Markus W Mahlberg Feb 26 '16 at 13:08
  • @MarkusWMahlberg, I've added the output of those commands as you requested. I've also created a new chain in IP tables allowing all of my servers and that port, which I could lock down a bit more later if I can get this working for security measures. But for now, even like this doesn't work. Also, I can log in from the App Server to the config server via mongo -u root -p xxx --authenticationDatabase admin hostOfConfigServer:27017 Thank you for the help. – Sam Feb 26 '16 at 15:06
1

Solved. I upgraded MongoDB from 3.2.1 to 3.2.3 and it magically started working. I didn't even need to recreate / reconfigure anything. It just started splitting the chunks correctly and works fine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.