5


I have a collection with documents similar to the following:

{
 _id: 'p_123456',
 id: 123456,
 kind: 'person'
 data: [...]
}

id field can contain either positive or negative integers.
This collection contains close to 100 million documents and I have a python script which I use in order to process the data from the collection.

I'm trying to export all the data, but into 8 different processes using the $mod operator in the following way:
mongoexport -u user -p password -d db -c collection --query "{\$and:[{kind:'person'},{id:{\$mod:[8,\$i]}}]}" | python process.py - where $i is a number between 0 - 7.

For some reason I've noticed that when I use this method with 8 processes not all the data is being exported just 65 million out of 87 million for this specific kind.

If I run a single mongoexport process with the query {kind:'person'} only, all 87 million documents are being exported.

Is it possible that running 8 different proccesses with $mod:[8,0] to $mod:[8,7] isn't enough in order to export all the data? what am I missing here?

EDIT 1:
Following AdamC suggestion, I ran the following map reduce:

var map = function() {
  if (this.kind == 'person'){
      var key = this.id % 8
      if (key < 0){
          key = key * (-1)
      }
      emit( key, 1 );
  }
};


var reduce = function( key, values ) {    
  return Array.sum(values);
}

The results returned as expected ~87 million records:

> db.map_reduce_modulus.find()
{ "_id" : 0, "value" : 10886482 }
{ "_id" : 1, "value" : 10878131 }
{ "_id" : 2, "value" : 10881552 }
{ "_id" : 3, "value" : 10882586 }
{ "_id" : 4, "value" : 10886060 }
{ "_id" : 5, "value" : 10882565 }
{ "_id" : 6, "value" : 10886171 }
{ "_id" : 7, "value" : 10883563 }

Therefor I don't think this is a type issue

Thanks for advance

3
  • Whats the output if you run the query with .count() in mongo?
    – Antonios
    Commented Dec 24, 2014 at 22:48
  • 87107467 results Commented Dec 29, 2014 at 6:42
  • mapReduce uses JavaScript, so your assertion here is incorrect. e.g var string = "10"; string % 8 would return a numeric 2 even though the data is a string. That's just how JavaScript works. $type actually looks at the BSON type, which is a very different assertion.
    – Neil Lunn
    Commented Feb 5, 2015 at 0:52

1 Answer 1

1

I can only speculate without having access to the data, but I suspect that you have non-numeric data in the id field. Try using the $type operator to verify that some of the fields have not been passed in as strings or similar and that all the fields are homogeneous in terms of type.

3
  • I've added more information to the body of the question. is it possible that mongoexport changes the type of fields? Commented Dec 26, 2014 at 16:41
  • Hi, I even went ahead and checked the $type, even though I suspect it's not the issue: db.my_collection.find({$and: [{kind: 'person'},{id:{$type: 16}}]}).count(): 87071995 Commented Jan 8, 2015 at 6:08
  • @MenyIssakov I can try to reproduce, but would need to know what indexes you have on that collection - can you add that information to the question? Can you also comment on the lower/upper bound for the integer in the id field?
    – Adam C
    Commented Jan 8, 2015 at 12:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.