0

We have some problems regarding balancer configuration. Our application which is set up with secondaryPreferred often fails to read documents from the secondary when balancer( chunk migration ) is running. So I changed the Balancer setting to the appropriate setting as below, but I could observe the unexpected behavior.

use config
db.settings.update(
   { "_id" : "balancer" },
   { $set : { "_secondaryThrottle" : true ,
              "writeConcern": { "w": 6} } },
   { upsert : true }
)

->  { "_id" : "balancer",   "stopped" : false,   "_secondaryThrottle" : true,   "writeConcern" : { "w" : 6 } }

1. Environment

Our cluster is as below.

( 1 primary, 5 secondary, one arbiter ) * 6 masters on Sharded cluster on MongoDB 3.2.4 on Google Cloud Platform Mongod instance type is "n1-standard-16 (16 vCPUs, 60 GB memory) with 500GB persistent disk"

2. What did I expect?

I expect that the application with reading as secondaryPreferred can read from the Secondary-mongod without failing. I think that the Balancer can wait for all completion of chunk-migration because I set up "writeConcern:6". So my expectations are as described above.

3. Test conditions and results

I configured as above settings and restarted balancer and execute the application. The application( secondaryPreferred ) failed with high frequency to read from the secondary-mongod.


Please let me ask a question.

  • Q1. What is the meaning of writeConcern in balancer setting? It seems that writeConcern does not work as shown in the above test.
  • Q2. How can I read without failing when chunk migration occurred?
  • Q3. What is the purpose of _secondaryThrottle? I have read the official document of v3.2. And official document described that _secondaryThrottle is true as the default value and has the same meaning as writeConcern: 2. But we can configure settings as below. If I set up _secondaryThrottle:false, does it makes the writeConcern invalid right? Is "_secondaryThrottle" the flag for enabling writeConcern?

e.g. { "_id" : "balancer", "stopped" : false, "_secondaryThrottle" : false, "writeConcern" : { "w" : 6 } }

sincerely

0

Nowadays, balancer doesn't honor _secondaryThrottle, nor w: when w:<=1. But, this case that w:6 means that when data is removed (range deleter at end of chunk move), we are removing documents one by one AND every document delete must have 6 acknowledge. If moved chunk had 100.000 documents, then there is 100.000 individual deletes with acknowledges. (slooooow)

If your query at secondary hits documents in the range what is under range delete, there will be a collision.

| improve this answer | |
  • Thank you very much for answering. I deduce that this situation is the same situation as below. SERVER-30600 It seems to solve this problem in mongodb v3.6. In the mean time, I would like to apply the workaround. However, I don't know an effective and validated workaround. Is there any idea? – umiyosh Sep 13 '17 at 6:17
  • No secondaryPreferred read? ;-) Read everything from Primary, as it "should" be done. – JJussi Sep 13 '17 at 15:23
  • Thank you. In terms of the workload of our system, we thought that it was difficult to adapt that workaround. But we decided to adopt reading everything from the primary. I will do it in the nighttime zone with less influence. Let me comment on the result here. – umiyosh Sep 14 '17 at 11:41
  • It always comes to the resources. If Primary node has "enough" memory and fast (high IOPS) data store, it can serve very high query load. But what then, when primary is moved to some other node what don't have same amount resources... Normally answer is "problems". But if you use node priority, you can decide where that primary is transferred in that situation and you can give that node same resources (as long as it is NOT at same DC than your preferred Primary). So, you need "beef up" only two nodes of those six. :-) – JJussi Sep 14 '17 at 13:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.