Sharding standalone MongoDB

Question

I've got a standalone MongoDB server with a lot amount of data on it. Let's say, it's srv1 server. I've decided to make a sharded cluster, so I've created additional servers srv2 and srv3.

We have:

• srv1 - existing mongodb server with data, that will be shard0001.
• srv2 - new mongodb server, that will be shard0000.
• srv3 - also new mongodb server, that will be config server with running mongos instance. Mongos instance running only on that server.
• vk - database I want to be sharded.

On srv3 I've started a mongos with --configdb rs0/127.0.0.1:27002 option and a mongod with --replSet rs0 --configsvr --dbpath /data/config --port 27002. After that, via the config instance i've added srv1 and srv2 as a shards with sh.addShard("srv1-2:27017") command. Also, on srv2 I've created an empty database and collections with the same names as on srv1. After that on config server i've executed sh.adminCommand({enableSharding: "vk"}); command.

What I have now:

Surprisingly, it's working. When i'm trying to read/write something to my database, using srv3 as connection point, it's working perfectly! When i've did something before (using srv1 standalone), average response time was about 2 seconds, now it's 0.02 seconds.

The problem I have:

Well, the sharding is not working. Database on srv2 is empty. Here is the output of the db.printShardingStatus() command from srv3:

--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("5d81f91bdde81f72ac4d1a0a")
  }
  shards:
        {  "_id" : "shard0000",  "host" : "srv2:27017",  "state" : 1 }
        {  "_id" : "shard0001",  "host" : "srv1:27017",  "state" : 1 }
  active mongoses:
        "3.6.3" : 1
  autosplit:
        Currently enabled: yes
  balancer:
        Currently enabled:  yes
        Currently running:  no
        Failed balancer rounds in last 5 attempts:  5
        Last reported error:  Connection refused
        Time of Reported error:  Wed Sep 18 2019 14:41:13 GMT+0200 (CEST)
        Migration Results for the last 24 hours:
                No recent migrations
  databases:
        {  "_id" : "config",  "primary" : "config",  "partitioned" : true }
                config.system.sessions
                        shard key: { "_id" : 1 }
                        unique: false
                        balancing: true
                        chunks:
                                shard0000       1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shard0000 Timestamp(1, 0)
        {  "_id" : "shard0000",  "primary" : "shard0000",  "partitioned" : true }
        {  "_id" : "vk",  "primary" : "shard0001",  "partitioned" : true }
        {  "_id" : "vk:",  "primary" : "shard0000",  "partitioned" : true }

Additional info:

The balancer says Connection refused, but i've opened 27017 and 27002 port on every machine in cluster. It's still not working. In mongos (srv3) log there are a lot of successful connections both to srv1 and srv2. Telnet says the same - everything is fine and opened. I've added shards without --replSet options. Mongod instances on srv1 and srv2 working without that options too. I didn't created any chunk keys or something like this.

Thank you for the reading, will be very grateful for any help.

Answer 1

About MongoDB 3.6 shards, from the official release notes:

Starting in 3.6, shards must be replica sets.

When you add shards, should be something like:

sh.addShard( "rs1/mongodb3.example.net:27017,mongodb4.example.net:27017,mongodb5.example.net:27017" )

With a status like:

--- Sharding Status ---
sharding version: {
   "_id" : 1,
   "minCompatibleVersion" : 5,
   "currentVersion" : 6,
   "clusterId" : ObjectId("5be0a488039b1964a7208c60")
}
shards:
   {  "_id" : "rs0",  "host" : "rs0/mongodb0.example.net:27017,mongodb1.example.net:27017,mongodb2.example.net:27017",  "state" : 1 }
   {  "_id" : "rs1",  "host" : "rs1/mongodb3.example.net:27017,mongodb4.example.net:27017,mongodb5.example.net:27017",  "state" : 1 }
active mongoses:
   "3.6.8" : 1

Answer 2

The problem I have:

Well, the sharding is not working. Database on srv2 is empty.

The main components of sharded cluster are the config server (stores meta data), mongos (routing to shards), and the shards (the distributed data storage).

The shards and and the config servers need to be replica sets. mongos allows access to the shards and the data.

A sharded cluster has multiple shards - data is distributed among the shards. Of these shards, one of them is a primary shard.

A MongoDB database has collections with data as documements. Sharding is about distributing these documents among shards. A sharded database can have collections with the data sharded and collections with data that is not sharded. Though sharding is enabled at database level, the actual sharding happens at each collection level.

Sharding a database does not distribute data among shards. Each collection need to sharded individually. All non sharded collections are located on the primary shard. All access to all collections happen via a mongos, irrespective of a collection is sharded or not.

First, setup the config server. Then, setup the mongos. Next, set up individual shards (this also includes converting any existing replica sets as a shard servers).

At this point, you have a sharded cluster configured. Now, it is time to distribute (or shard) a sharded database's collection data among the shards.

Sharding the data:

An important concept in sharding the data is the shard key. This determines how the collection data is distributed among the shards in a cluster. So, determine the shard key on your collection to be sharded. Determining a shard key is an important design aspect of the data and the application that accesses the sharded data.

Shard the collection based upon the shard key. After this step you will be able see the data is distributed among the shard servers (it may take some time to verify the data distribution).

Also, see Deploy a Sharded Cluster.