2

We are seeing around 800 open connections to our Mongodb. To clarify our issue:

  1. We have 5 different clients who write the data into Mongodb. Each of these clients - write four different type of documents into their respective collections. And they have 25 goroutines each. So in total at anytime there could be around 100 goroutines from each of these clients. Due to high load, we see that 500 connections from these clients are opened (we confirmed this by stopping all these clients and the connection count reduces)

  2. We have 3 different servers, which read the Mongodb primary. And these are serving different requests. And these have around 40 connections open.

The remaining 260 connections (800 - (500+40)), seems to be leaks. We are trying to reproduce the issue here.

In total, we have around 800 connections opens.

Due to this huge opened connections, we see that normal queries take around 10 seconds! But when we stop the 5 clients, the response of these queries are served between 400-800 ms.

I have already verified using explain and all these queries are served on Index only .

Any tips on debugging/handling this issue.

Note: My guess is, due to huge number of writes, the mongodb is constantly busy and not able to process the query. We have primary and a secondary. But currently we are always reading only from the primary. Any tips here would also be good.

We are using mongodb 3.0 with MMAPv1. We are planning to move to wiredTiger.

Update1:

All the details of the mongo data, cpu, etc.

We have two DB within mongo

DB1  17.945GB
DB2  17.945GB

DB1 Stats in GB

rs1:PRIMARY> db.stats(1024*1024*1024)
{
    "db" : "DB1",
    "collections" : 15,
    "objects" : 4171674,
    "avgObjSize" : 2214.681749340912,
    "dataSize" : 8.604424327611923,
    "storageSize" : 13.75045008957386,
    "numExtents" : 138,
    "indexes" : 34,
    "indexSize" : 1.5781951248645782,
    "fileSize" : 17.9296875,
    "nsSizeMB" : 16,
    "extentFreeList" : {
        "num" : 0,
        "totalSize" : 0
    },
    "dataFileVersion" : {
        "major" : 4,
        "minor" : 22
    },
    "ok" : 1
}

DB2 Stats in GB

rs1:PRIMARY> db.stats(1024*1024*1024)
{
    "db" : "DB2",
    "collections" : 36,
    "objects" : 2383787,
    "avgObjSize" : 6308.773688253187,
    "dataSize" : 14.005948513746262,
    "storageSize" : 16.816604614257812,
    "numExtents" : 315,
    "indexes" : 70,
    "indexSize" : 0.6485644727945328,
    "fileSize" : 17.9296875,
    "nsSizeMB" : 16,
    "extentFreeList" : {
        "num" : 0,
        "totalSize" : 0
    },
    "dataFileVersion" : {
        "major" : 4,
        "minor" : 22
    },
    "ok" : 1
}

Currently, we have reduced our 5 clients to each have only 60 connections each. So the current mongostat of our server looks like:

insert query update delete getmore command flushes mapped vsize  res faults idx miss % qr|qw ar|aw netIn netOut conn set repl     time
    *0     2      4     *0       0    12|0       0  40.1G 83.3G 5.7G      0          0 1|189 1|189    2k    13k  279 rs1  PRI 17:22:59
    *0    *0      3     *0       0    16|0       0  40.1G 83.3G 5.7G      0          0 1|192 1|192    1k    13k  279 rs1  PRI 17:23:00
     8     3    812     *0      64    60|0       0  40.1G 83.3G 5.7G      1          0 1|190 1|203  702k   458k  279 rs1  PRI 17:23:05
     1     1    208     *0      49   103|0       0  40.1G 83.3G 5.6G      1          0 1|193 1|213  217k   134k  280 rs1  PRI 17:23:08
    44     2   2030     *0     218    82|0       0  40.1G 83.3G 5.7G      4          0 1|209 2|211    2m     1m  280 rs1  PRI 17:23:09
     1    *0     96     *0      12    15|0       0  40.1G 83.3G 5.7G      1          0 1|211 1|211   77k    71k  280 rs1  PRI 17:23:10
    10     1   1057     *0     116    98|0       0  40.1G 83.3G 5.7G      0          0 1|209 2|215  941k   609k  280 rs1  PRI 17:23:13
    *0     2     26     *0       6    35|0       0  40.1G 83.3G 5.7G      2          0 3|217 3|217   26k    43k  280 rs1  PRI 17:23:14
    *0    *0      1     *0       0    12|0       0  40.1G 83.3G 5.7G      0          0 3|218 3|218    1k    13k  280 rs1  PRI 17:23:15
    *0     1      2     *0       0    12|0       0  40.1G 83.3G 5.7G      0          0 3|220 3|220    1k    12k  280 rs1  PRI 17:23:16
insert query update delete getmore command flushes mapped vsize  res faults idx miss % qr|qw ar|aw netIn netOut conn set repl     time
    *0    *0     *0     *0       0    11|0       0  40.1G 83.3G 5.7G      0          0 3|220 3|220  806b    12k  280 rs1  PRI 17:23:17
     4     1    660     *0      64    28|0       1  40.1G 83.3G 5.7G      1          0 1|219 2|221  583k   394k  280 rs1  PRI 17:23:19
    28     5   1775     *0     148    34|0       0  40.1G 83.3G 5.7G      5          0 0|200 0|207    2m     1m  280 rs1  PRI 17:23:20
     1     1    429     *0      10    11|0       0  40.1G 83.3G 5.7G      5          0 0|189 0|189  420k   309k  280 rs1  PRI 17:23:21

Notice the conn count => 280.

At this time, the top output looks like

top - 17:24:01 up 6 days,  3:07,  1 user,  load average: 8.55, 6.31, 5.96
Tasks:  94 total,   1 running,  93 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.3 us,  2.0 sy,  0.0 ni, 91.8 id,  0.7 wa,  0.0 hi,  0.2 si,  0.0 st
KiB Mem:   8176828 total,  8042296 used,   134532 free,   121640 buffers
KiB Swap:        0 total,        0 used,        0 free.  6569776 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 2039 root      20   0 83.338g 5.714g 4.795g S  28.2 73.3   6233:19 mongod
 5741 root      20   0       0      0      0 D   1.0  0.0  21:23.15 kworker/u8:0
 1035 deploy    20   0  425500  11724   4728 S   0.3  0.1  50:39.10 consul
    1 root      20   0   33504   2472   1120 S   0.0  0.0   0:11.16 init
    2 root      20   0       0      0      0 S   0.0  0.0   0:00.10 kthreadd

We do notice that the Cached memory has reached to almost 8GB of the total RAM. How to handle this is the question here.

Regarding the explain, what we see is, when we call the explain sometimes that itself takes 2-3 seconds to respond. This only means that mongodb is not responding (ie, we are not able to open the session at this time or we are connected, but has not yet started the processing).

Just by stopping our clients, we close the session, and then mongodb is lightning fast.

  • 1
    That's odd. What is your total database + index size, what is the total available RAM, and what do your performance metrics show re: memory/cpu/diskio/hard faults? If your hypothesis is correct, the question would be 'why'? Almost always a MongoDB performance issue is indexing. Is it indexed AND sorted? That's critical. Please share your explain.pretty() output if you can as well. Thanks. – Ali Razeghi Oct 23 '15 at 20:28
  • @AliRazeghi I have updated the question. See Update 1. Please let me know, if you have any other questions – Sundar Oct 23 '15 at 21:28
  • Your problem doesn't seem to be the high numbers of connections but the high number of concurrent updates. From the mongostat i guess you use MMap that uses collection level locking. If under normal load an update takes 400-800 ms then its normal under high write to get poor performance, since your updates are queuing waiting to be served. WT will help and sharding will also increase the write scope. Can you share the updates you are running and the explain for these updates. – Antonios Oct 23 '15 at 22:42
  • @Antonios Thanks for the info. WT will be done over the weekend. Sharding - we will check it. How to get explain for updates. At least using mgo the go library, it seems not possible to get explain on Update. WRT update: We collect performance data from browsers, and we have indeed a huge number of write calls. So it definitely is write intensive. One question would be - even if we read from Secondary in this case, it would result in the same situation - is that correct? Because, secondary is constantly syncing with the primary. Is there a way to allow read as higher priority? – Sundar Oct 23 '15 at 23:01
  • Use the query part of the update to run an explain. The query part of the update should always use an index. Read from the SEC won't help much as primary CRUD operations will also execute on the secondaries. – Antonios Oct 24 '15 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.