1

I'm using Locustio service as load testing tool for my app. Without details, I'm using the same aggregation query with simulated 100 users, in this case RPS (requests per second) is about 20. In first 30 seconds every query time is about 10-50ms, but then (when about 65 users active) it increases to 500-1000ms. I'm using MMAPv1, so query is cached for sure, so the question is: why query execution time is increasing?

Of course, it's only test case, in real I'm doing different queries, but in that case I wasn't sure that all the queries was cached. That's why I decided to test It only with one query. Sorry for my bad explanation.

I think the problem is in connection number, still it's only 100. How cna I optimize the performance of my app?

UPDATE1

db.serverStatus about connections number:

    "connections" : {
        "current" : 20,
        "available" : 51180,
        "totalCreated" : NumberLong(32)
    },

All of my engine settings are defaults This is my db.collection_name.findOne() part, which involved in aggregate query:

{
    "_id" : ObjectId("571764be1e31c8556e80b589"),
    "ppl_ids" : [
        19,
        1,
        20,
        21,
        22,
    ],
}

The pipeline of aggregate query itself:

{$match: {ppl_ids: {$in: [16,1245,592] } } }

The part of db.collection_name.getIndexes():

    {
        "v" : 1,
        "key" : {
            "ppl_ids" : 1
        },
        "name" : "ppl_ids_1",
        "ns" : "dbname.collection_name",
        "background" : false
    },
0

Update based on MongoEngine usage in comments:

MongoEngine defers to pymongo for the connection pool setting. Hence, based on the pymongo FAQ:

In versions before 2.6, the default max_pool_size was 10, and it did not actually bound the number of open connections; it only determined the number of connections that would be kept open when no longer in use.

Starting with PyMongo 2.6, the size of the connection pool is capped at max_pool_size, which now defaults to 100. When a thread in your application begins an operation on MongoDB, if all other connections are in use and the pool has reached its maximum, the thread pauses, waiting for a connection to be returned to the pool by another thread.

Assuming you are using a version later than 2.6 you will have a default of 100, which theoretically should be enough. However even if the responses are very quick you still have some overhead (connect, run query, return result, return connection to pool) with the pool in order to reach 100 concurrent users on the front end. Therefore, I would still look at increasing the max_pool_size parameter to see if that unblocks your load testing (say 150 to start with).

| improve this answer | |
  • Thank you for response, Adam! Added some details in my question. If you need more details, just tell me what to add, my settings are almost all defaults, the field is simple as pipeline of aggregation query – vadimb May 11 '16 at 16:26
  • What driver is your load tester using to connect/query MongoDB, and are you using defaults there? – Adam C May 11 '16 at 17:21
  • Adam, my load tester is imitating users behavior, it doesn't using anything to connect MongoDB. I just created some test cases, these only sending get requests with some URL-arguments, and then my app using these arguments, making queries and executes them with MongoDB. So connecting to database is app's work only, not load tester. – vadimb May 11 '16 at 17:30
  • OK, then what driver and settings does the app use? – Adam C May 11 '16 at 18:02
  • I'm using mongoengine, it is using Pymongo i guess, but i'm not sure how to get the connection limit information in it – vadimb May 11 '16 at 18:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.