0

Here is what I did to test sharding performance.

I have 4 servers, I ran config server and router on a server. 3 others are meant to be shards.

First, I only added one shard to router, then I ran a test where it inserted 10000 documents into a collection. I have enabled sharding on database and collection before.

What I see is that if I write on a mongos with more than one shard, the insert time is somewhat worse than single-shard. My inserts are one at a time, not batch inserts.

The process is this:

  1. I first drop database
  2. Then enabled sharding on database (named test)
  3. Then shard collection (named files) using key (md5, randomly generated)
  4. Insert 10000 documents

Then I run again my test (insert) which only clears collection and inserts 10000 documents into that. No drop happens here.

I ran insert test with 1,2 and 3 shards in cluster, and 3 insert tests on each (the process for 3 tests are above, 1 clean test then 2 remove-only test). The results are somehow the same. No big difference here. Even performance is better with one shard!

As for read test, I use the above inserted document. At first I have 3 shard, and test reading all documents. Then I remove one shard and again run test. Then I remove another shard and test again. Of course I wait for shards to be drained completely before running next test.

I ran this test 3 times on each number of shards, starting with 3 shards and decreasing that. Again no performance benefit here, and somehow 2 shards worked better than 3 or 1 shard. What?!

What I got from these tests is that the times for 1 shard or 3 shards do not differ, even sometimes I get better performance using 1 shard!

I am confused a bit, isn't sharding supposed to help performance? Am I doing the tests correctly? Why do I get the same results? Do I need to have any configuration?

Info:

MongoDB Version: 3.0.2

I run config server like mongod --configsvr --dbpath <blah> --bind_ip <blah> --port <blah>

I run router like mongos --configdb <blah_from_config_server> --bind_ip <blah>

Router and config server are on the same server (different ports).

I run shards like mongod --dbpath <blah> --bind_ip <blah>

Each shard is on different server.

Servers are CentOS 6.5 x64, and I have downloaded MongoDB binary just some hours ago. I run instances from this binary, not from the repos.

EDIT:

I just finished testing multi-process testing. 20 processes, each inserting 10 batches of 1000 document, totally 200000 documents. It takes ~10 seconds to perform this with 3 nodes, while ~7 seconds on 1 node

EDIT:

Please see the results below. With 3 shards, this test took ~2600 seconds while without sharding it took ~145 seconds! Really confused now...

Output of sh.status() for when I have tested insert using 10 processes each inserting 1000 batches of 1000 documents, resulting in a 10M documents collection.

--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("553db915382bcef5dece453e")
}
  shards:
        {  "_id" : "shard0000",  "host" : "192.168.10.20:27017" }
        {  "_id" : "shard0001",  "host" : "192.168.10.30:27017" }
        {  "_id" : "shard0002",  "host" : "192.168.10.40:27017" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : true,  "primary" : "shard0000" }
                test.files
                        shard key: { "_id" : "hashed" }
                        chunks:
                                shard0002       88
                                shard0001       88
                                shard0000       89
                        too many chunks to print, use verbose if you want to force print

Output of sh.status() for when I have tested insert using 10 processes each inserting 1000 batches of 1000 documents, resulting in a 10M documents collection. Shading was disabled here.

--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("553db915382bcef5dece453e")
}
  shards:
        {  "_id" : "shard0000",  "host" : "192.168.10.20:27017" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : false,  "primary" : "shard0000" }

UPDATE

I just finished another test with 1 shard (I have enabled sharding) and the same test took ~410 seconds!

--- Sharding Status ---
  sharding version: {
        "_id" : 1,
        "minCompatibleVersion" : 5,
        "currentVersion" : 6,
        "clusterId" : ObjectId("553db915382bcef5dece453e")
}
  shards:
        {  "_id" : "shard0000",  "host" : "192.168.10.20:27017" }
  databases:
        {  "_id" : "admin",  "partitioned" : false,  "primary" : "config" }
        {  "_id" : "test",  "partitioned" : true,  "primary" : "shard0000" }
                test.files
                        shard key: { "_id" : "hashed" }
                        chunks:
                                shard0000       266
                        too many chunks to print, use verbose if you want to force print
  • Please provide the output of sh.status() – Markus W Mahlberg Apr 28 '15 at 11:38
  • furthermore, I need the connection string. I noticed that --shardsrv option is missing in the command line. Have you activated it in the config file? – Markus W Mahlberg Apr 28 '15 at 11:41
  • I followed docs.mongodb.org/manual/tutorial/deploy-shard-cluster and I don't see anything about --shardsrv. – vfsoraki Apr 29 '15 at 4:47
  • Sry, I have changed shared key to {_id: hasshed} for testing, and the output is because of that. I will provide more result when this test is finished. – vfsoraki Apr 29 '15 at 4:49
  • Still, please provide the output of sh.status(). – Markus W Mahlberg Apr 29 '15 at 4:50
0

An important consideration with any benchmarking is that you make sure that you are testing the right thing (i.e. the bottlenecks really are where you think that they are). In this instance the code that is performing your inserts may be single-threaded and synchronous meaning that is is going to wait for each item to hit the data store before starting next. The following:

Send document 1 to the store
(wait for acknowledgement)
Send document 2 to the store
(wait for acknowledgement)
Send document 3 to the store
(wait for acknowledgement)
...

will likely be no slower than

Send document 1 to shard 1 of store
(wait for acknowledgement)
Send document 2 to shard 2 of store
(wait for acknowledgement)
Send document 3 to shard 3 of store
(wait for acknowledgement)
...

You will most likely not see any benefit here, in fact you might see a small drop in performance. The improvements you seek will likely show up in other access patterns (multiple clients accessing at once, and/or multiple threads and/or asynchronous access in a single client process).

The size of the documents and the way you are batching them up (if at all) to send to the store could be significant as well, as that will also dictate which bottlenecks are significant (local I/O on the shards, network bandwidth, network latency, CPU use, ...).

tl;dr: It is likely that the use pattern you are using as your test is not one that sharding offers benefits to.

  • I just finished testing multi-process testing. 20 processes, each inserting 10 batches of 1000 document, totally 200000 documents. It takes ~10 seconds to perform this with 3 nodes, while ~7 seconds on 1 node – vfsoraki Apr 27 '15 at 10:03
  • 1
    I do testing like this all the time and one very important consideration is does the use case warrant sharding yet? There is some overhead that comes with sharding that may actually slow down your results compared to a single shard. At the moment, what I believe you are demonstrating is that your use case is better served by a single shard. When you begin to tax that single shard then multiple shards may benefit you. 200,000 moderately sized documents is nothing for a single shard to process. Also; to test sharding correctly, use a proper 3 config, 3 router + shards topology. – SDillon Apr 27 '15 at 10:44
  • @thelastblack: you should edit your question to include the extra detail, comments sometimes get cut off (or otherwise ignored) so someone who might otherwise have useful advice may miss the relevance and skip on by without knowing they can help. – David Spillett Apr 27 '15 at 15:51
  • Thanks, I have added the info to question. Maybe you are right, this quantity is nothing for a single shard. I will test with more data, and add results to question if it helped or not. – vfsoraki Apr 27 '15 at 16:17
  • When you run the load test on a single shard the index on {md5:1} existed? I would recommend you to perform the same test using {_id:hashed}. Did you tried to run the load tests with 2 mongos using just round robin? – Antonios Apr 27 '15 at 20:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.