We have the following environment:
- 3 servers + 3 replica set servers
- Each server has 3 shards
- Our main collection has around 40,000,000 documents that average at around 6kb
- Our shard key is hashed(_id) - _id being pure BsonID
- We peak daily at around 25,000 I/OPS and low point at around 10,000
We want to run an ETL that loads all of the documents on the main collection, do some in memory calculation (in our application tier) and then dumps it into an external DB.
We took the very poor and naive approach and simply loaded documents without a query using limit, skip and batchSize - which was a complete failure (severly hurt our service level - Even though we set the readPreference to secondary).
I was wondering what would be the best approach to be able to load all of our documents without hurting the performance of our database.
The data can be a bit stale (a few seconds delay from the actual data on the primary is fine).
Sorry in advance if this is the wrong part of StackExchange to post my question.