1

We have in environment deployed some really small DBs (5GB) as directory per DB.Under dirs is LVM with XFS, where we have implemented logic for "brake" when DB space should be full-filled (all is due to some strict disk quotas - not really interesting at all I think...). Our solution runs on MongoDB v3.4.19 on CentOS 7.5. Everything looks ok.

Some of DBs has more than 60% used disk space by single collection and in our scenario we encountered that, while restoring 2.3GB dump (all into 5GB database dir/=disk, the preallocation after 2GBs asks OS for another space in (I think) power of 2 strategy (or something very similar).

This situation hits ours triggers, and eventually should lead to lock of DB (which is still better than fall of whole cluster because of no space left). Basically same situation we have with 10GB DB. (And in some way we can say that similar situation can occur with literally any capacity...)

My point of question is: Am I able to influence steps (maximum size lower than ~2GB) of (pre)allocation growing of Mongo/WT for restoring (only) or at all?

(Performance impact is not highest prioritized in this case.)

I've found some promising possibilities through db.adminCommand and configStrings for storage engines, but I was not able to affect or list some of possible parameters nor change something to turn things to good... Especially file_extend parameter of WT through 

db.adminCommand( { setParameter: '1', wiredTigerEngineRuntimeConfig: { <parameter> } } )

looked promising but I didn't find way to pass it to cluster... Some of tries ended with error 22 - Invalid argument, but I wasn't able to find more about...

The other found, seemed suitable for MMAP engine only - newCollectionsUsePowerOf2Sizes=false.

This makes restoring of dumps (eventually backups) into cluster much more bothersome...

Hope that issue is clearly explained and any hint/tip would be appreciated... If some additional info needed, let me know.

Update:
I've done screenshots where (I believe) is clearly visible what I'm talking about above... There are $ ls output (upper left), $ df output (upper right) and $ mongorestore progress (bottom left). (Better once to see than thousand times to hear (read)...)

Beginning Beginning First alloc (to 1.4GB - used 650MB) First_step_up Second alloc (to 2.4GB - used 1.1GB) Second_step_up Third alloc (to 4.4GB - used 2.1GB) Third_step_up At the restore end (still allocated 4.4GB - used 2.3GB) Restore_end Back to normal (extra space deallocated) Space_freed

Thank you.

Zdenek

Improve this question
6
  • Just trying to understand your scenario, Are you doing mongorestore to restore your data? In that process, it preallocates space and asks for more space? Totally 2+3 = 5 GB of data you are restoring to 5 GB of LVM?
    – Mani
    Apr 9, 2019 at 20:18
  • @Mani Yes, we are running mongorestore from gzipped dump. One specific example could be ->2,2GB dump (real DB size about ~9GB). When restoring collection with about 2,1GB size (the biggest one in DB) I can see, that after consuming first 2GB of space on storage process asks OS for allocating next 2GB. But really uses, let's say 200-300MB in file (which would be fair enough). As we have set triggers at 80% of storage capacity used, it hits them, and in some cases reach even 90% when we lock DB for writes. I would like set smaller step of preallocation, to get rid of this...
    – Zdenek M.
    Apr 10, 2019 at 7:50
  • 2,2GB dump (real DB size about ~9GB) - Are you trying to say, you have 2 gzip files with each file size equal to 2 GB? Also 2,1GB size - Is it again 2 gzip files with size equal to 1 GB each? If yes, what is the actual size of the data in the second case? When you are restoring a compressed gzip fle , first it will de-compress the gzip file, so restotration process will restore the data with actual size of the data(not the compressed size), in that case it may require more space than the compressed file.
    – Mani
    Apr 10, 2019 at 15:30
  • @Mani I believe that all misunderstands came from my usage of decimal comma instead of point. I'm sorry, my fault... I have 21 files in whole dump (total size 2.2GB, largest collection has 2.0GB of it - gzipped). Mentioned largest collection has storage size 2.2GiB (and data size 8.4 GiB). I'll provide some detailed info from specific restore as update to main question for clarification what I mean exactly. Thanks for interest anway, even if I don't make it easier to understand as I see. :)
    – Zdenek M.
    Apr 11, 2019 at 13:51
  • Try creating an issue official MongoDB Jira tracker jira.mongodb.org. Also, Please post the response or Issue details here for us also to track :)
    – Mani
    Apr 11, 2019 at 19:26

2 Answers 2

Reset to default
1

WiredTiger does not preallocate data files. Only journal files are pre-allocated (at 100MB per pre-allocation), see journaling process.

The disk exhaustion you're seeing is likely due to these journal files. Journal files are optimized for quick writes, and their contents will be persisted in a more permanent manner on a WiredTiger checkpoint which occurs every ~60 seconds. Typically, once persisted in data files, you should see less disk used, since now the journal files can be removed. Note that the journal files are only removed by WiredTiger following a checkpoint and at no other times.

I would suggest you to allow for some temporary extra space during this restore/data load process. Once the database is in a steady state, you would see a more accurate size.

The power-of-2 preallocation is specific to MMAPv1 storage engine, which is deprecated. This is not relevant to WiredTiger.

Improve this answer
5
  • For the journal files, MongoDB creates a subdirectory named journal under the dbPath directory. So the preallocation of journal files won't create any data in database directories. Also, mongorestore doesn't use any journaling mechanism so the size increase of database directory is not because of the journal files pre-allocation.
    – Mani
    Apr 11, 2019 at 15:25
  • I'm not sure, what to say, I need to look at it in more detail and try a few tests... Because it doesn't sound bad, but as you can see in screens in question update, there's no extra file/dir created at DBdir (which is dir per DB configured). But as you exceed datafile size, "some allocation is ordered" at the expense of space for DB itself (as seen at $ df output beside). All I need is make smaller steps in consuming (to be more accurate to real storage usage). Or force data persistence more often (if it turn out to be true) and if "allocation" get influenced by desired manner...
    – Zdenek M.
    Apr 11, 2019 at 15:49
  • The journal directory is actually under the dbpath, so you should see the dbpath increasing in size as more journal is allocated. mongorestore does use journal. It doesn't dump the data straight into WT data files, but is using bulk insert method in the Go driver. If your checkpoints are taking a long time (due to hardware, disk speed, etc.), then WT will use more journal, meaning that the size of your dbpath would be increasing.
    – kevinadi
    Apr 12, 2019 at 0:11
  • One thing you can try is doing the restore using a faster disk and a slower disk, and compare the size of the dbpath and the journal directory between different disk specs. Something like while true; do du -s; du -s journal; sleep 1; done > ~/out.txt will show you the size of the directories per second as the restore is progressing.
    – kevinadi
    Apr 12, 2019 at 0:13
  • @KevinAdistambha It's actually true about some journaling while mongorestore, but we have dbDir configured to /srv/data/mongo and as could be seen in screenshots (in $ df part) it inflicts consuming space where expected... I'll try some of suggested.
    – Zdenek M.
    Apr 12, 2019 at 8:52
0

Am I able to influence steps (maximum size lower than ~2GB) of (pre)allocation growing of Mongo/WT for restoring (only) or at all?

As per the Google group forum here Each data file will never be more than 2GB. We use a signed 32bit int to index inside of each file so we cant use more than 2^31 bytes. We have a soft limit of 16000 files per db [1] which limits you to 32TB in a single DB per process. It is mostly just used as a sanity check, so you could easily change that number and recompile to raise that limit. That said, you will soon run into the 48bit virtual address space limitation of current x86_64 cpus, of which only 47 bits is available to user space, leading to a hard limit of just under 128TB. Also, if you plan to use durability, we double-map every file so you are limited to around 64TB.

I mean a single file in the file system. A database in MongoDB spans multiple files, so this doesn't impose a limit on data set size. Yes this applies to 64-bit OSs as well. We use the same data-file format on all platforms so you can freely move your files around.

The 32-bit offset is mostly a space optimization. It allows us to point to anywhere within a single db using a 64-bit DiskLoc stuct with a 32-bit file number and a 32-bit offset into that file. This is what mongo uses internally when storing pointers to disk.

For further your ref here, here and here

Improve this answer
1
  • 1
    This explains about what is the limitations of 32 bit operating system over the 64 bit operating system. Also it talks about Mmapv1 storage engine, not wired tiger. If my understanding is correct the question is , is there any preallocation happening while restoring the data using mongorestore and if yes , how to avoid that.
    – Mani
    Apr 10, 2019 at 14:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.