0

I created a small speed test

from misc.Database import Database
import time

db = Database.getDb()


def main():
    test_db = db.test_db.find({})
    return "done"

if __name__ == '__main__':
    start = time.time()
    for i in range(10000):
        main()
    end = time.time()
    print(end - start)

where db is my pymongo client. When monitoring the mongod log, I realized it opened 2 connections when I run the test. When I run Robo3T, it opened 25 connections to the Mongod. Why is the connection not opened per request? How many connections will open each time you query the database?

0

First, you are using an old method to connect. The newer and better method is using MongoClient, which is supported in all recently released supported drivers. It's explained in detail here: https://mongodb.github.io/node-mongodb-native/driver-articles/mongoclient.html. Note that although the link discusses the node driver implementation of it, it is also relevant for pymongo.

Second, you're not iterating on the returned cursor. This means that the find() query was not executed on the server. I presume you see a really fast result in this test, which will not be the case if you're actually getting data from the server.

Third, Python is single-threaded by its nature. Pymongo is smart enough to realize you're not iterating the cursor, so it reuses the connection instead of creating a new one. Creating a new connection is very expensive, so drivers will not do it unless it's really necessary. The two connections you're seeing is likely one for monitoring server status, and the other for executing your query. The cost of opening a new connection leads to drivers using a connection pool instead.

Fourth, Robo3T is a GUI, so it naturally needs to open more connections since presumably it requires a lot of information from the server, perhaps asynchronously. This is a very different situation vs. a driver. You can't really compare the two.

Finally, performance testing is a tricky subject and must be done in a very controlled manner. Some things that needs planning:

  • How is the server provisioned, how big are the documents, how compressible are they, etc.
  • Can you determine whether any performance bottleneck is due to the server, the network, the driver, the query, or how the language is used. For example, if you try to iterate on the cursor using list(db.test_db.find()), how can you tell if the database is slow, or is it Python's list() method that is slow?
  • Running the testing code in the same machine as the database server might introduce resource contention, artificially skewing the result.
  • Hi, sorry for the late comment, but I was re-routed to other priorities. Thank you for your reply, I am using the latest pymongo client, just that I have abstracted it away in a class called Database, so db is actually already the client. You are right, iterating through the cursors make it much slower, although the connections are still just 2 open connections, so I'm not quite sure how the connections operate... Regarding performance testing, I think I will try to investigate more using system profiler, but I just want to know the basics of pymongo connections – Lumo Woong Jul 9 '18 at 2:38
  • It was explained in the answer above: Pymongo is smart enough to realize you're not iterating the cursor, so it reuses the connection instead of creating a new one. Creating a new connection is very expensive, so drivers will not do it unless it's really necessary. This is also true when you're iterating the cursor. It will do it in a single thread, so Pymongo will not create a new connection. – kevinadi Jul 9 '18 at 2:41
  • Ok, correct me if I'm wrong, connections will open whenever you request for a cursor from the db, but since Python is single-threaded, there will generally be very little connections per thread, so we can safely assume number of connections will be roughly equal to 2 * no. of users who are accessing the db currently. This is all based on pymongo, and excludes some background application or cron jobs accessing the db – Lumo Woong Jul 9 '18 at 4:09
  • I don't think you can make that generalized assumption. Basically pymongo will not open a new connection unless it has to. It will reuse any existing connection in the connection pool first. See api.mongodb.com/python/current/…. It can be think of like you have a connection pool, where you have minimal control over the content of the pool and even how the connections inside the pool are used as it will be opened, closed, and utilized as required by the application. – kevinadi Jul 9 '18 at 4:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.