0

I have a field in my collection which is a String in nature. When i run aggregation for this field in data it takes considerable time in aggregation. Total 6 million records are in collection which has almost 500k different value for this field.

I am in impression if a field is numeric with similar scenario then aggregation on that field will be very fast.

So i planned to save the string as unique integer value by converting it java biginteger but mongo does not support that. Could you please suggest me what could be the best possible way for this.

This is the data we want to aggregate

{
    "_id" : ObjectId("54d7ac5c3848e23f6f471b9f"),
    "articleId" : [
            {
                    "articleId" : "17084694",
                    "articleCount" : 1
            }
    ],
    "host" : "abc.com",
    "count" : NumberLong(1),
    "cat" : "test test",
    "catIds" : [
            "-2128958273",
            "2886704",
            "5912641"
    ],
    "timeStamp" : NumberLong("1423420500000"),
    "interval" : 1,
    "tags" : "magnetic fields"
}

{
    "_id" : ObjectId("54d7ac5c3848e23f6f471b9f"),
    "articleId" : [
            {
                    "articleId" : "17084694",
                    "articleCount" : 1
            }
             {
                    "articleId" : "17084695",
                    "articleCount" : 1
            }
    ],
    "host" : "abc.com",
    "count" : NumberLong(1),
    "cat" : "test test",
    "catIds" : [
            "-2128958273",
            "2886704",
            "5912641"
    ],
    "timeStamp" : NumberLong("1423420500000"),
    "interval" : 1,
    "tags" : "magnetic fields"
}

{
    "_id" : ObjectId("54d7ac5c3848e23f6f471b9f"),
    "articleId" : [
            {
                    "articleId" : "17084695",
                    "articleCount" : 1
            }
             {
                    "articleId" : "17084696",
                    "articleCount" : 1
            }
    ],
    "host" : "abc.com",
    "count" : NumberLong(1),
    "cat" : "test test",
    "catIds" : [
            "-2128958273",
            "2886704",
            "5912641"
    ],
    "timeStamp" : NumberLong("1423420500000"),
    "interval" : 1,
    "tags" : "taps records"
}

Result will be

{
    "_id" : ObjectId("54d7ac5c3848e23f6f471b9f"),
    "articleId" : [
            {
                    "articleId" : "17084694",
                    "articleCount" : 2
            }
             {
                    "articleId" : "17084695",
                    "articleCount" : 1
            }
    ],
    "host" : "abc.com",
    "count" : NumberLong(2),
    "cat" : "test test",
    "catIds" : [
            "-2128958273",
            "2886704",
            "5912641"
    ],
    "timeStamp" : NumberLong("1423420500000"),
    "interval" : 1,
    "tags" : "magnetic fields"
}

{
    "_id" : ObjectId("54d7ac5c3848e23f6f471b9f"),
    "articleId" : [
            {
                    "articleId" : "17084695",
                    "articleCount" : 1
            }
             {
                    "articleId" : "17084696",
                    "articleCount" : 1
            }
    ],
    "host" : "abc.com",
    "count" : NumberLong(1),
    "cat" : "test test",
    "catIds" : [
            "-2128958273",
            "2886704",
            "5912641"
    ],
    "timeStamp" : NumberLong("1423420500000"),
    "interval" : 1,
    "tags" : "taps records"

} As you see the aggregation is on tags field where two things are added one is count second artcileCount for same tags. IT takes a lot of time. Hash i was referring to keep tag string as some integer form and aggregate that

  • what is the range of data in your case? Maybe you can use ObjectId, numberLong? depending your data range. Maybe you can use index to speedup the aggregation. Or you can use mapreduce.... – BAE Feb 5 '15 at 16:37
  • It would have been useful to share the aggregation, the indexes for that collection and a sample document. Also please correct your question you are referring to hash and mongo perfectly supports that. Which hash algorithm you are using? – Antonios Feb 6 '15 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.