0

We are currently trying to get to the bottom of some high background flush times on our MongoDB installation. As part of this investigation we have been looking at the $set operation we perform on an array embedded in one of our document types. The array contains 348 empty documents to begin with, and over the course of a week these array element will have sub-documents inserted (if empty) and then subsequently updated (if they already exist). The sub-documents are approximately 100 bytes in size and are not indexed.

So my question is, when the document is flushed to disk, what actually gets written, the entire document or just the sub-document that has been updated?

  • What specific version of MongoDB and O/S are you using, and what sort of storage config (file system & disk type)? Also, how long are your average flush times? Flush times are only part of the overall picture so in isolation it's hard to guess what the problem might be. Are you using a monitoring service like MMS? If so, are there any other metrics that appear to correlate with your long flush times? – Stennie Jun 16 '15 at 13:05
  • We are running 3.0.3. Most of our problems seemed to start when we upgraded to 3.0 but it could just be coincidence as our flush times looked to have been increasing exponentially over the past few months. Going by MMS, our flush times track network, opcounters and page faults (Windows). Curiously, even overnight when the input to our system is very low we still get background flush times of a second or two. We have read that any times over a second or so are bad, so its concerning that even when almost idling we exceed these times. At the peak times of day it's as high as 30 seconds. – King Roger Jun 17 '15 at 7:20
  • Just to add, the system is a replica set running in Azure on Windows 2012 R2 Standard A3 VM instances with 4 striped drives for the data folder. Claimed performance is 500 IOPS per disk. – King Roger Jun 17 '15 at 7:51
1

The array contains 348 empty documents to begin with, and over the course of a week these array element will have sub-documents inserted (if empty) and then subsequently updated (if they already exist). The sub-documents are approximately 100 bytes in size and are not indexed.

One consideration with this use case is that your documents are consistently growing. MongoDB used a record allocation or padding strategy to allow documents to grow in-place. For example, if your document starts off as 1000 bytes MongoDB 2.6 or newer will round this up to a 1024 byte record allocation for MMAP (as per the Power of 2 Size default strategy). Updates that don't grow the size of the document beyond the current record allocation are more efficient for the server to execute.

However, if you added 100 bytes to a document which was initially 1000 bytes, the document would have to be moved to a new record allocation in storage (and associated index entries would also have to be updated). So in this example, the next allocation for a 1100 byte document would be 2048 bytes (allowing for ~9 more 100 byte fields to be added before a new record allocation was needed for this document). Indexes in MongoDB include the storage location of the document, so a document move will result in an update for every index entry referencing that document.

You can check the frequency of document moves by looking at the nmoved value for slow updates (or by enabling increased levels of logging / system profiling). Frequent document moves can definitely have a performance impact. Common strategies include either reconsidering the data model (eg. moving the growing portion of the document to a separate collection if appropriate) or adding manual padding to the initial document allocation. The default power of 2 allocation strategy is designed to avoid the need for manual padding in most cases, but if your documents start small and grow quickly you might be able to avoid some initial document moves.

So my question is, when the document is flushed to disk, what actually gets written, the entire document or just the sub-document that has been updated?

The answer will depend on the size of your document and the nature of updates since the last background flush. I'll assume you are using a default configuration with MMAP storage engine and journal enabled.

By default data changes are written twice: once to fast append-only journal files (committed to disk every 100ms) and again to a private view in memory (flushed to data files every 60s). The background flush process is a periodic asynchronous write of all pages that have been "dirtied" in memory since the last flush. Journal commit and background flush intervals can be influenced by both server configuration and write concerns. For a good overview of the process see How MongoDB’s Journaling Works.

The MMAP storage engine will fetch the full document into memory before applying updates. The standard x86 page size is 4KiB so a single document may be represented by one or more pages -- or multiple documents may be part of a single page in memory.

So, if you are updating a single document the writes will include:

  • all changes written to the journal
  • all changes written to the oplog (if that node is part of replica set)
  • any pages dirtied for that document since the last background flush

An important caveat is "since the last background flush". Multiple updates affecting the same pages within a given sync interval will effectively be batched.

If you're trying to get to the bottom of performance issues then consistently high background flush times (particularly as a large or increasing percentage of the default 60s flush interval) are definitely of concern, but should be reviewed in the context of other metrics such as page faults, I/O stats, and lock percentage. I would also review the MongoDB Production Notes for general tips and upgrade to the latest MongoDB production release for your major version (i.e. latest 2.6.x or 3.0.x if there's a newer x than your current version).

  • Hi Stennie, thanks for the reply. We are aware of the growing document issue, but the doc only grows to ~16-18KB so we predict it will move only 3-5 times in a week. We might pad to get a bit better performance but we don't think its the cause of our high flush times. So, given our document is generally larger than a single page, I guess you are saying if a 100 bytes of data is added it will require a single 4K page to be updated (or two if the data straddles a page?) BUT the pages for the entire document must be read into memory first. Is that correct? – King Roger Jun 16 '15 at 12:20
0

On Windows, make sure you've allocated sufficient page file space per the recommendations here.

From the notes:

Configure Windows Page File For MMAPv1 Configure the page file such that the minimum and maximum page file size are equal and at least 32 GB. Use a multiple of this size if, during peak usage, you expect concurrent writes to many databases or collections. However, the page file size does not need to exceed the maximum size of the database.

A large page file is needed as Windows requires enough space to accommodate all regions of memory mapped files made writable during peak usage, regardless of whether writes actually occur.

The page file is not used for database storage and will not receive writes during normal MongoDB operation. As such, the page file will not affect performance, but it must exist and be large enough to accommodate Windows’ commitment rules during peak database use.

NOTE Dynamic page file sizing is too slow to accommodate the rapidly fluctuating commit charge of an active MongoDB deployment. This can result in transient overcommitment situations that may lead to abrupt server shutdown with a VirtualProtect error 1455.

I cannot explain it in terms of internals, but our operational experience is that background flush times change proportionately with the size of the page file. The recommendation is 32GB, but in our experience making sure the page file is the same size as physical memory has a dramatic effect on background flush times. In our case (a write heavy application), we saw flush times fall from 15 seconds to under 2 seconds.

  • Hi, thanks for the reply. On Windows we had to make our pages files about 60 GB for a 200 GB database with only about 7 GB of physical memory. The page file gradually increased to this size over time as we would get a 'virtual alloc' type exception every now and again that could only be fixed by increasing the page file - I think this is/was a known bug in MongoDB. We've now switched to Linux and the performance is considerably better although the background flush is creeping higher again. Its been suggested by the 10Gen guys that Azure storage disk latency might be an issue. – King Roger Sep 18 '15 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.