We have the following environment:

  • 3 servers + 3 replica set servers
  • Each server has 3 shards
  • Our main collection has around 40,000,000 documents that average at around 6kb
  • Our shard key is hashed(_id) - _id being pure BsonID
  • We peak daily at around 25,000 I/OPS and low point at around 10,000

We want to run an ETL that loads all of the documents on the main collection, do some in memory calculation (in our application tier) and then dumps it into an external DB.

We took the very poor and naive approach and simply loaded documents without a query using limit, skip and batchSize - which was a complete failure (severly hurt our service level - Even though we set the readPreference to secondary).

I was wondering what would be the best approach to be able to load all of our documents without hurting the performance of our database.

The data can be a bit stale (a few seconds delay from the actual data on the primary is fine).

Sorry in advance if this is the wrong part of StackExchange to post my question.


  • Any reason why you don't use the aggregation framework instead of doing this in your application logic? Please provide a sample document and the expected output. – Markus W Mahlberg May 7 '15 at 20:36
  • @MarkusWMahlberg we're extracting some fields of our users into Redis. i.e. a user looks like { _id: '12345', Name: 'Marky Mark', Age: 45 }, we'd like to take the _id and Age and store them in Redis for every player – Ron May 7 '15 at 23:54
  • You can make this a covered query in MongoDB, effectively cashing the values in RAM – Markus W Mahlberg May 8 '15 at 5:39
  • Simply create a compound index over them and use protection to limit the returned fields to _id, name and age: db.collection.ensureIndex({_id:1, name:1,age:1});db.collection.find(query,{name:1,age:1}). No need for another DBMS and added complexity. – Markus W Mahlberg May 8 '15 at 5:48
  • Mark, we need to move it to Redis as Mongo can't handle the type of queries we're attempting to do; the example I gave you was just that, an example. Besides my question was how to run an ETL on a live production environment without hurting the quality of service. – Ron May 8 '15 at 9:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.