5


I have a collection with documents similar to the following:

{
 _id: 'p_123456',
 id: 123456,
 kind: 'person'
 data: [...]
}

id field can contain either positive or negative integers.
This collection contains close to 100 million documents and I have a python script which I use in order to process the data from the collection.

I'm trying to export all the data, but into 8 different processes using the $mod operator in the following way:
mongoexport -u user -p password -d db -c collection --query "{\$and:[{kind:'person'},{id:{\$mod:[8,\$i]}}]}" | python process.py - where $i is a number between 0 - 7.

For some reason I've noticed that when I use this method with 8 processes not all the data is being exported just 65 million out of 87 million for this specific kind.

If I run a single mongoexport process with the query {kind:'person'} only, all 87 million documents are being exported.

Is it possible that running 8 different proccesses with $mod:[8,0] to $mod:[8,7] isn't enough in order to export all the data? what am I missing here?

EDIT 1:
Following AdamC suggestion, I ran the following map reduce:

var map = function() {
  if (this.kind == 'person'){
      var key = this.id % 8
      if (key < 0){
          key = key * (-1)
      }
      emit( key, 1 );
  }
};


var reduce = function( key, values ) {    
  return Array.sum(values);
}

The results returned as expected ~87 million records:

> db.map_reduce_modulus.find()
{ "_id" : 0, "value" : 10886482 }
{ "_id" : 1, "value" : 10878131 }
{ "_id" : 2, "value" : 10881552 }
{ "_id" : 3, "value" : 10882586 }
{ "_id" : 4, "value" : 10886060 }
{ "_id" : 5, "value" : 10882565 }
{ "_id" : 6, "value" : 10886171 }
{ "_id" : 7, "value" : 10883563 }

Therefor I don't think this is a type issue

Thanks for advance

  • Whats the output if you run the query with .count() in mongo? – Antonios Dec 24 '14 at 22:48
  • 87107467 results – Meny Issakov Dec 29 '14 at 6:42
  • mapReduce uses JavaScript, so your assertion here is incorrect. e.g var string = "10"; string % 8 would return a numeric 2 even though the data is a string. That's just how JavaScript works. $type actually looks at the BSON type, which is a very different assertion. – Neil Lunn Feb 5 '15 at 0:52
1

I can only speculate without having access to the data, but I suspect that you have non-numeric data in the id field. Try using the $type operator to verify that some of the fields have not been passed in as strings or similar and that all the fields are homogeneous in terms of type.

  • I've added more information to the body of the question. is it possible that mongoexport changes the type of fields? – Meny Issakov Dec 26 '14 at 16:41
  • Hi, I even went ahead and checked the $type, even though I suspect it's not the issue: db.my_collection.find({$and: [{kind: 'person'},{id:{$type: 16}}]}).count(): 87071995 – Meny Issakov Jan 8 '15 at 6:08
  • @MenyIssakov I can try to reproduce, but would need to know what indexes you have on that collection - can you add that information to the question? Can you also comment on the lower/upper bound for the integer in the id field? – Adam C Jan 8 '15 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.