I have a collection with documents similar to the following:
{
_id: 'p_123456',
id: 123456,
kind: 'person'
data: [...]
}
id field can contain either positive or negative integers.
This collection contains close to 100 million documents and I have a python script which I use in order to process the data from the collection.
I'm trying to export all the data, but into 8 different processes using the $mod operator in the following way:
mongoexport -u user -p password -d db -c collection --query "{\$and:[{kind:'person'},{id:{\$mod:[8,\$i]}}]}" | python process.py
- where $i is a number between 0 - 7.
For some reason I've noticed that when I use this method with 8 processes not all the data is being exported just 65 million out of 87 million for this specific kind.
If I run a single mongoexport process with the query {kind:'person'}
only, all 87 million documents are being exported.
Is it possible that running 8 different proccesses with $mod:[8,0]
to $mod:[8,7]
isn't enough in order to export all the data? what am I missing here?
EDIT 1:
Following AdamC suggestion, I ran the following map reduce:
var map = function() {
if (this.kind == 'person'){
var key = this.id % 8
if (key < 0){
key = key * (-1)
}
emit( key, 1 );
}
};
var reduce = function( key, values ) {
return Array.sum(values);
}
The results returned as expected ~87 million records:
> db.map_reduce_modulus.find()
{ "_id" : 0, "value" : 10886482 }
{ "_id" : 1, "value" : 10878131 }
{ "_id" : 2, "value" : 10881552 }
{ "_id" : 3, "value" : 10882586 }
{ "_id" : 4, "value" : 10886060 }
{ "_id" : 5, "value" : 10882565 }
{ "_id" : 6, "value" : 10886171 }
{ "_id" : 7, "value" : 10883563 }
Therefor I don't think this is a type issue
Thanks for advance
var string = "10"; string % 8
would return a numeric2
even though the data is a string. That's just how JavaScript works.$type
actually looks at the BSON type, which is a very different assertion.