7

We're attempting to better optimize how we're using our MongoDB instance. We seem to be routinely getting high lock-percentages, and are looking to help minimize that. Here is some mongostat output:

insert  query update delete getmore command flushes mapped  vsize    res faults locked % idx miss %     qr|qw   ar|aw  netIn netOut  conn       time 
     1    107    186      0       0     196       0  3.06g   7.3g   333m      0     11.2          0       0|0     2|0    66k   224k    85   15:55:22 
     2    102    285      0       0     296       0  3.06g   7.3g   333m      0     15.7          0       0|0     2|0    89k   216k    84   15:55:23 
     2     79    325      0       0     335       0  3.06g   7.3g   333m      0     20.2          0       0|0     3|0    96k   149k    85   15:55:24 
     2     92    193      0       0     203       0  3.06g   7.3g   333m      0     10.9          0       1|1     6|1    63k   149k    86   15:55:25 
     3    102    235      0       0     245       0  3.06g   7.3g   331m      0     14.5          0       0|0     2|0    75k   177k    84   15:55:26 
     3     79    267      0       0     275       0  3.06g   7.3g   331m      0     16.5          0       1|0     2|0    80k   133k    86   15:55:27 
     2     66    219      0       0     226       0  3.06g   7.3g   264m      0     14.3          0       0|0     2|0    66k   112k    88   15:55:28 
     2    100    201      0       0     211       0  3.06g   7.3g   334m      0     10.2          0       0|0     3|0    67k   142k    87   15:55:29 
     3    118    227      0       0     244       0  3.06g   7.3g   322m      0     13.8          0       3|1     6|1    78k   150k    87   15:55:30 
     2    112    189      0       0     198       0  3.06g   7.3g   334m      0     10.8          0       0|1     2|2    64k   213k    87   15:55:31 
     2     80    266      0       0     278       0  3.06g   7.3g   246m      0     15.8          0       0|1     3|1    82k   179k    86   15:55:32 
     1     82    307      0       0     314       0  3.06g   7.3g   334m      0     18.1          0       0|0     2|0    89k   158k    86   15:55:33 
     2     94    278      0       0     285       0  3.06g   7.3g   334m      0     17.1          0       0|0     0|0    83k   184k    86   15:55:34 
     3    101    246      0       0     256       0  3.06g   7.3g   332m      0     14.2          0       0|0     1|0    82k   179k    86   15:55:35 
     3     99    203      0       0     213       0  3.06g   7.3g   334m      0     12.5          0       0|0     2|0    67k   154k    88   15:55:36 
     2    115    174      0       0     189       0  3.06g   7.3g   335m      0       11          0       1|0     3|0    63k   172k    88   15:55:37 
     2     97    199      0       0     209       0  3.06g   7.3g   335m      0     10.3          0       0|0     2|0    65k   192k    87   15:55:38 
     2    103    366      0       0     373       0  3.06g   7.3g   334m      0     23.5          0       1|4     3|4   107k   256k    85   15:55:39 
     2    105    338      0       0     349       0  3.06g   7.3g   334m      0     22.9          0       0|0     1|0   101k   207k    83   15:55:40

This is a lot better than it used to be, thanks to better indexing. However, we clearly have more to do. Things about this data-set:

  • Hardware is a 4-proc box, load-average is generally between 1.3 and 1.9
  • 4GB of RAM
  • The SAN-backed storage is reporting latencies peaking at 35ms, but generally between 5m and 20ms most of the time.
  • I/O operations are very low
  • The 'qr' and 'qw' numbers do suggest we're not suffering big queuing.

We're using Mongo to track meta-data as documents pass through our processing platform. A Mongo Document is created for each actual document we have (actual documents are ye olde Office-type files). Each processing stage queries some information, and then writes information back (some times quite a bit of it). Depending on the data we're working with, there can be many stages.

This is an update-heavy workload, so lock-percentage is a key scaling statistic. We haven't sharded yet, in large part because we need to see how far a single instance can scale before we need to shard.

What other areas do we need to investigate to reduce lock-percentage, or have we just hit the wall and need to shard?

Improve this question

1 Answer 1

Reset to default
3

Those are interesting stats. I think you might be suffering from document size growth in your updates, where the document needs to be copied to a new spot on the disk. If this is indeed the case, you might be able to recover some of that lock percentage by manually padding your documents. It adds a bit more complexity on the first insert, but isn't too bad. See this document from the official docs: http://www.mongodb.org/display/DOCS/Padding+Factor#PaddingFactor-ManualPadding

Just an idea...

Improve this answer
4
  • And not a bad one at that, since our Document sizes do indeed grow considerably from their first insert. The longer processing paths do yield larger end-documents. We do have some intelligence on how much to pad since one of the first stages is to determine the type of file.
    – sysadmin1138
    Apr 23, 2012 at 23:52
  • Hi from Fairfax, by the way. Ha.
    – Eve Freeman
    Apr 24, 2012 at 1:02
  • After running a test set or three and watching the padding-factor, it's pretty clear that we're getting a lot of growth. A recent run started at a PF of 1.3 and by the end it was at 1.89. So manual-padding looks highly likely to be in our future.
    – sysadmin1138
    Apr 24, 2012 at 15:23
  • Sounds promising!
    – Eve Freeman
    Apr 24, 2012 at 15:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.