1

When prototyping a query I've been using just the Mongo Shell to make sure my aggregation query is working fine:

# Default query
    query = [{
        # Supposedly forces MongoDB to initially use this index.
        '$sort': {
            '_id': 1
        }
    }, {
        '$unwind': '$things.titles',
    }, {
        '$group': {
            '_id': '$things.titles.name',
            'count': {'$sum': 1},
        }
    }, {
        '$sort': {'count': -1}
    }]

Being run on this collection which the top of .stats() look like this:

...
"count": 4065493,
"size": 31275790798,
"avgObjSize": 7692,
"storageSize": 11493429248,
...

When I run this query in Mongo Shell, it finishes within five minutes tops. But when I ran it as part of the request handler in a Flask route via PyMongo it took all night to complete.

How can I either:

  1. Speed this up considerably
  2. Determine what's causing the slowdown?

This is also running on a AWS instance with 4GB of RAM which MongoDB takes up a maximum of 42% I've noticed.

EDIT:

For some clarification, when I run this script:

import time
import subprocess

start_time = time.time()
subprocess.call(["mongo", "localhost:27017/ars-xbox", "/Users/simius/Downloads/test.js"])
print("--- %s seconds ---" % (time.time() - start_time))

I get a run of about 105 seconds which is an acceptable length.

However, when I run the exact same query as part of a Flask route, it runs for up to 20 minutes or more.

  • 2
    forty-two – Max Vernon Apr 12 '16 at 18:07
  • We'll need more info in this case. First, is PyMongo running off the server or the same machine you're running in the shell? Can you use .Explain to show us the output on both? Also, lol Max. Good to see you in a funny mood today. docs.mongodb.org/manual/reference/operator/meta/explain – Ali Razeghi Apr 12 '16 at 18:10
  • Hey @AliRazeghi the PyMongo script I just added in the edit is running on my local machine with a ssh tunnel from to the mongodb server running on AWS. The Flask server can be run on either my local machine or on the server (I tried both) and both result in the same slowdown. – dalanmiller Apr 18 '16 at 3:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.