0

I've written small application in Python (I used Twisted). Program receives data and writes it to MongoDB. Every 2 days MongoDB stops responding:

2015-02-11T10:47:20.918+0100 [initandlisten] can't create new thread, closing connection

I don't know what is going on because we have ~10 concurrent connections which is very small amount. Just before it crashed mongostat had return that output:

insert  query update delete getmore command flushes mapped  vsize    res        locked db idx miss %     qr|qw   ar|aw  netIn netOut  conn repl       time
    10     *0      5     *0       0    11|0       0  5.03g  17.7g  2.22g           .:0.3%          0       0|0     0|0    10k     4k    11  PRI   10:43:55
    12      2      6     *0       0    14|0       0  5.03g  17.7g  2.22g           .:2.1%          0       0|0     0|0    12k   232k    11  PRI   10:43:56
    14      1      7     *0       0    16|0       0  5.03g  17.2g  2.21g  foo:0.9%          0       0|0     0|0    15k   230k    11  PRI   10:43:57
    10      1      5     *0       0    11|0       0  5.03g  17.7g  2.22g           .:2.0%          0       0|0     0|0     9k     4k    11  PRI   10:43:58
    12     *0      6     *0       0    13|0       0  5.03g  17.7g  2.22g  foo:0.6%          0       0|0     0|0    13k     4k    11  PRI   10:43:59
    14      1      7     *0       0    15|0       0  5.03g  17.7g  2.22g           .:2.3%          0       0|0     0|0    14k     5k    11  PRI   10:44:00

Load average: 0.49, 0.42, 0.32

Free RAM: 12 GB

OS: FreeBSD

Free disk space: 76 GB

Application looks like this:

class Device(protocol.Protocol):

    def __init__(self, factory):
        self.db = factory.db

    def dataReceived(self, data):
        deferred = threads.deferToThread(self.handle, data)
        deferred.addCallback(self.on_success)
        deferred.addErrback(self.on_error)

    def on_success(self, response):
        if response is not None:
            self.transport.write(response)

    def on_error(self, failure):
        logging.error('Error in deferred: %s' % failure.getErrorMessage())

    def handle(self, tracker, input):
        self.db.buffer.insert({ ''' writing data ''' })

class Rpc(jsonrpc.JSONRPC):

    def jsonrpc_get_by_id(self, id, limit=1000):

        deferred = threads.deferToThread(self.get_by_id, id, limit)
        deferred.addCallback(lambda result: json.JSONDecoder().decode(json.dumps(result, default=json_util.default)))
        deferred.addErrback(lambda failure: logging.error(str(failure), exc_info=True))

        return deferred

    def get_by_id(self, id, limit=1000):

        client = pymongo.MongoClient()
        result = []

        db = client.foo

        for doc in db.record.find({'_id': {'$gt': ObjectId(str(id))}}).limit(limit):
            result.append(doc)

        return result

class Server(protocol.Factory):

    def __init__(self):
        self.client = pymongo.MongoClient()
        self.db = self.client.foo

    def buildProtocol(self, addr):
        return Device(self)

reactor.listenTCP(6969, Server())
reactor.listenTCP(6970, server.Site(Rpc())) # JSON RPC
reactor.run()

I realized that this situation occurs when db.serverStatus().connections.totalCreated reaches high number like 300k.

Any idea what is the reason of that behavior?

  • It's pretty true of any application using any database server back-end that you should only establish a connection once or better yet work from a "pool" of connections. What you are doing is establishing a connection in an "instance" method. That is bad news and not good design. At the most basic, look for a "singleton" or "app global" pattern. Generally something better. – Neil Lunn Feb 11 '15 at 11:15
  • I tried to use pool of connections (invoke pymongo.MongoClient() in Device class) but effect was the same. – Bald Feb 11 '15 at 11:28
  • I think the point here is that it needs to be "shared" and a true singleton or global. It's really not a DBA question anyhow. Your application scoping is wrong, hence why you have so many connections building up and not being released. – Neil Lunn Feb 11 '15 at 11:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.