I have a big collection (over 400 millions records) which I want to divided between 4 shards. I created infrastructure (4 shards, config server, mongos) and all work well (balancer is working and data is being moved). It has being worked greater than 15 hours and working now. My shards have 2662, 77, 77, 77 chunks. It is too slow!
As I know balancer try to keep mongo available. But now nobody use db and my main task is to finish balancing asap.
Could anybody know any command which set balancing as the most task?
1 Answer
My shards have 2662, 77, 77, 77 chunks.
If you are creating a new sharded collection and want to minimise the time to re-balance a large amount of data, the recommended approach would be to pre-split chunks in an empty collection before inserting the data. In your example, you would have pre-split based on the data distribution so each shard would have an equal number of chunks representing about 1/4 of the data.
Assuming you have chosen a suitable shard key for data distribution going forward, there should not be a significant number of regular chunk migrations unless you are adding or removing shards.
Could anybody know any command which set balancing as the most task?
If your shards are backed by replica sets you can make the balancer more aggressive by disabling _secondaryThrottle
, which reduces the write concern used for documents migrating between shards. By default, _secondaryThrottle
is true
which is equivalent to a {w:2}
write concern: each document move during chunk migration propagates to at least one secondary before the balancer proceeds with the next document. In MongoDB 3.0+, there is also an option to configure an explicit writeConcern
for the secondaryThrottle
operation.
For example, to disable _secondaryThrottle
from a mongos
shell:
db.getSiblingDB('config').settings.update(
{ "_id" : "balancer" },
{ $set : { "_secondaryThrottle" : false } },
{ upsert : true }
)
Balancer setting updates will not take immediate effect if there is currently a balance round / migration in progress, so you may want to disable & re-enable the balancer.
For normal usage the default _secondaryThrottle
behaviour is recommended to ensure documents have propagated and reduce the impact of balancing. If your replica sets have more than 3 members, you could also consider increasing the writeConcern
to majority
(see: Write Concerns for Replica Sets).
-
So I should backup my collection, clear it, create chunks in an empty collection, let balancer to balance it between my shards, restore data (it will be inserted in different shards). Am I right? Can I use mongodump/mongorestore? Commented Oct 19, 2015 at 9:00