We are in the process of moving our MongoDb replication sets to new hardware but is encountering a problem when adding our new replication set to the cluster.

2017-03-28T20:07:18.434+0200 W NETWORK  [ReplicaSetMonitorWatcher] No primary detected for set mongoset004
2017-03-28T20:07:28.442+0200 I NETWORK  [ReplicaSetMonitorWatcher] node mongosrv014:27018 believes it is primary, but its election id of 7fffffff0000000000000005 and config version of 3 is older than the most recent election id 7fffffff000000000000002c and config version of 71238
2017-03-28T20:07:28.443+0200 I NETWORK  [ReplicaSetMonitorWatcher] node 192.168.1.93:27018 believes it is primary, but its election id of 7fffffff0000000000000005 and config version of 3 is older than the most recent election id 7fffffff000000000000002c and config version of 71238

We ended up in this state after removing the old replication set by following the guide to remove shards from a cluster. Then we replaced the servers with our new ones (keeping their names and ip addresses) and configured the "new" MongoDb set on these servers and tried to add them to the cluster once more

sh.addShard('mongoset004/mongosrv014:27018,mongosrv015:27018,mongosrv016:27018')

We've probably missed something crucial when removing the original cluster but can't find any documentation that points us in the right direction.

Update 2017-04-05

The output from db.getSisterDb('config').shards.find()

mongos> db.getSisterDB('config').shards.find()
{ "_id" : "mongoset001", "host" : "mongoset001/192.168.1.85:27018,192.168.1.86:27018,192.168.1.87:27018" }
{ "_id" : "mongoset002", "host" : "mongoset002/192.168.1.91:27018,192.168.1.92:27018,mongosrv010:27018" }
{ "_id" : "mongoset003", "host" : "mongoset003/mongosrv019:27018,mongosrv020:27018,mongosrv021:27018" }
{ "_id" : "mongoset005", "host" : "mongoset005/mongosrv022:27018,mongosrv023:27018,mongosrv024:27018" }
mongos> sh.addShard('mongoset004/mongosrv014:27018,mongosrv015:27018,mongosrv016:27018')
mongos> db.getSisterDB('config').shards.find()
{ "_id" : "mongoset001", "host" : "mongoset001/192.168.1.85:27018,192.168.1.86:27018,192.168.1.87:27018" }
{ "_id" : "mongoset002", "host" : "mongoset002/192.168.1.91:27018,192.168.1.92:27018,mongosrv010:27018" }
{ "_id" : "mongoset003", "host" : "mongoset003/mongosrv019:27018,mongosrv020:27018,mongosrv021:27018" }
{ "_id" : "mongoset005", "host" : "mongoset005/mongosrv022:27018,mongosrv023:27018,mongosrv024:27018" }
{ "_id" : "mongoset004", "host" : "mongoset004/mongosrv014:27018,mongosrv015:27018,mongosrv016:27018" }

The output from sh.status() after adding the set

mongos> sh.status()
--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("582480867c09909734754b86")
}
  shards:
        {  "_id" : "mongoset001",  "host" : "mongoset001/192.168.1.85:27018,192.168.1.86:27018,192.168.1.87:27018" }
        {  "_id" : "mongoset002",  "host" : "mongoset002/192.168.1.91:27018,192.168.1.92:27018,mongosrv010:27018" }
        {  "_id" : "mongoset003",  "host" : "mongoset003/mongosrv019:27018,mongosrv020:27018,mongosrv021:27018" }
        {  "_id" : "mongoset004",  "host" : "mongoset004/mongosrv014:27018,mongosrv015:27018,mongosrv016:27018" }
        {  "_id" : "mongoset005",  "host" : "mongoset005/mongosrv022:27018,mongosrv023:27018,mongosrv024:27018" }
  active mongoses:
        "3.2.10" : 1
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  0
        Migration Results for the last 24 hours:
                1 : Failed with error 'aborted', from mongoset001 to mongoset004
  databases:
        {  "_id" : "emil_prod",  "primary" : "mongoset001",  "partitioned" : true }
                emil_prod.hpCurrentState
                        shard key: { "_id" : "hashed" }
                        unique: false
                        balancing: true
                        chunks:
                                mongoset001       13
                                mongoset002       13
                                mongoset003       13
                                mongoset005       13
                        too many chunks to print, use verbose if you want to force print
  • 1
    What sh.status() says? You can use command getSisterDB('config').shards.find() too to see your shard configuration. – JJussi Apr 4 '17 at 14:11
  • @JJussi Thanks, it didn't get us any further. I've run the commands you asked for and updated the question. – m__ Apr 5 '17 at 8:10
  • Ok.. Can you stop everything. All mongod processes (shards, configs) and all mongos. Check with 'netstat -anp |grep 2701' that there is no connections to config servers from anywhere. Then start first all shards, then all config servers and last one mongos. Connect to that mongos and check situation. Because it looks like that there is/was mongo process (mongod or mongos) what didn't get changes of structure of cluster. And during that "start" process. Follow log files (mongod.log and mongos.log) for any errors. – JJussi Apr 5 '17 at 8:31
  • Well, it's a production system so stopping everything must be a last resort. There's nothing else we could try first? – m__ Apr 5 '17 at 10:10
  • Because mongoset004 is stll empty, you could remove it from cluster (use admin; db.runCommand( { removeShard: "mongoset004" } ); #repeat until it's removed). Check errors from config-server mongod.logs. Then recreate mongoset004, first as single node replica set, add it to cluster and if there is no errors (including balancing), then you start adding secondaries to that mongoset004, one by one. – JJussi Apr 5 '17 at 10:57

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.