We have in environment deployed some really small DBs (5GB) as directory per DB.Under dirs is LVM with XFS, where we have implemented logic for "brake" when DB space should be full-filled (all is due to some strict disk quotas - not really interesting at all I think...). Our solution runs on MongoDB v3.4.19 on CentOS 7.5. Everything looks ok.
Some of DBs has more than 60% used disk space by single collection and in our scenario we encountered that, while restoring 2.3GB dump (all into 5GB database dir/=disk, the preallocation after 2GBs asks OS for another space in (I think) power of 2 strategy (or something very similar).
This situation hits ours triggers, and eventually should lead to lock of DB (which is still better than fall of whole cluster because of no space left). Basically same situation we have with 10GB DB. (And in some way we can say that similar situation can occur with literally any capacity...)
My point of question is: Am I able to influence steps (maximum size lower than ~2GB) of (pre)allocation growing of Mongo/WT for restoring (only) or at all?
(Performance impact is not highest prioritized in this case.)
I've found some promising possibilities through db.adminCommand and configStrings for storage engines, but I was not able to affect or list some of possible parameters nor change something to turn things to good... Especially file_extend parameter of WT through
db.adminCommand( { setParameter: '1', wiredTigerEngineRuntimeConfig: { <parameter> } } )
looked promising but I didn't find way to pass it to cluster... Some of tries ended with error 22 - Invalid argument, but I wasn't able to find more about...
The other found, seemed suitable for MMAP engine only - newCollectionsUsePowerOf2Sizes=false.
This makes restoring of dumps (eventually backups) into cluster much more bothersome...
Hope that issue is clearly explained and any hint/tip would be appreciated... If some additional info needed, let me know.
Update:
I've done screenshots where (I believe) is clearly visible what I'm talking about above... There are $ ls output (upper left), $ df output (upper right) and $ mongorestore progress (bottom left). (Better once to see than thousand times to hear (read)...)
Beginning
First alloc (to 1.4GB - used 650MB)
Second alloc (to 2.4GB - used 1.1GB)
Third alloc (to 4.4GB - used 2.1GB)
At the restore end (still allocated 4.4GB - used 2.3GB)
Back to normal (extra space deallocated)
Thank you.
Zdenek
2,2GB dump (real DB size about ~9GB)
- Are you trying to say, you have 2 gzip files with each file size equal to 2 GB? Also2,1GB size
- Is it again 2 gzip files with size equal to 1 GB each? If yes, what is the actual size of the data in the second case? When you are restoring a compressed gzip fle , first it will de-compress the gzip file, so restotration process will restore the data with actual size of the data(not the compressed size), in that case it may require more space than the compressed file.