MongoDB multi-datacenter deployment and shard keys

Question

I am trying to check my understanding after reading docs and some older questions, so my question is quite loaded:

1. In a sharded cluster, each shard should (and as of 3.6 must?) reside on its own replica set. 2) Each replica set has its own master.

So suppose I wanted to deploy a Mongo cluster in 2 availability zones. So I set up a 6-node cluster with 2 shards on 2 separate replica sets, such that master for each set is in different datacenter. Slave nodes are distributed such that at least one slave from each replica set is in each datacenter, so:

1. If I use "majority" read concern, would my mongod instances within same replica set have to communicate between my datacenters to verify that data is available on at least 2 instances? It sounds a little silly. Or would replication take longer, as each instance needs to confirm to other instances that its write is complete?
2. If I use "available" read concern, can I guarantee that my application reads from closest instance? Can I make my application (say Mongo CLI) prefer a certain slave?
3. If I wanted to guarantee each write happens to the closest master, I could use region as shard key.

Now suppose I wanted to do same exercise with 4 shards, 2 on each replica set.

1. Is it possible to have more than one shard in one replica set? Ie each node would duplicate 2 sets of data?
2. If so, and I wanted to still make sure each write happens to the closest master, I would use a compound shard key, with region and some other attribute. Can I somehow configure ranges of my shard key hash results to each master? Ie I'd like to achieve something like:

Shard 1 and 2 reside in US-West datacenter and shard 3 and 4 reside in US-East datacenter

Then for my compound shard key of (region, foo) I would want to guarantee that:

| region = US-West | region = US-East | 
| foo < 0 | foo >=0| foo < 0 | foo >=0|
| shard 1 | shard 2| shard 3 | shard 4|

Answer 1

Yes, your question is quite loaded...

1.1 Yes, every shard should have RS 1.2 Every RS have it's own master.

2.1. Only if you do slave reads. Normally read is done from primary. It will not affect replication directly. Only write of data, when write concern is majority. 2.2. If you do slave read AND you connect to specific instance, then data is read from that instance. 2.3. Yes you can do that.

3.1. NO. 3.2. NO.

You can have (of course) 2 shards in US-West and 2 shards in US-East... You just have then 4 shard cluster what uses compound sharding key {region:1,foo:1}, but every shard have different data. You just need to set chunk limits by hand (to have that foo < 0 and foo >= 0) and disable balancing between shards... End result may not be the best available, because of uneven distribution of data.

Best possible sharding key distributes written data evenly to every shard, so there is no need for balancing between shards and queries never need to read from multiple shards. With "bad" key, you are writing all data to one shard and then balancing is moving that data to other shards (later).