9

I recently compacted my collection using the command:

 db.<collectionName>.runCommand( "compact" )

And now my collection size seems to be bigger than size on disk!

SECONDARY> db.<collectionName>.stats()
{
"ns" : "<databaseName>.<collectionName>",
"count" : 2937359,
"size" : 5681676492,                   # 5.6 GB
"avgObjSize" : 1934.2805874256433,
"storageSize" : 4292853728,            # 4.2 GB
"numExtents" : 2,
"nindexes" : 2,
"lastExtentSize" : 2146426864,
"paddingFactor" : 1.669999999836597,
"flags" : 1,
"totalIndexSize" : 220735648,
"indexSizes" : {
    "_id_" : 162326304,
    "e_1_" : 58409344
},
"ok" : 1

}

I don't understand how this is possible. Aren't all mongodb collections backed-by-disk at all times?

Can anyone explain these results?

  • I've seen stats like that before, but don't have an explanation. Try running a validate? – Eve Freeman Apr 28 '12 at 2:15
6

storageSize is the sum of all extents for that data, excluding indexes.

So that collection takes up 2 extents, they are ~2GB each, hence ~4GB. size includes indexes and I believe a couple of other things which inflate the number. Neither really represents the proper on-disk size. For disk size, db.stats() has a filesize field which is closer to what you want I think you're looking for.

The manual is somewhat better at outlining what the various fields mean, see here for collections:

http://docs.mongodb.org/manual/reference/collection-statistics/

And here for database stats:

http://docs.mongodb.org/manual/reference/database-statistics/


Some other potentially relevant information:

The compact command does not shrink any datafiles; it only defragments deleted space so that larger objects might reuse it. The compact command will never delete or shrink database files, and in general requires extra space to do its work, usually a minimum of one extra extent.

If you repair the database it will essentially rewrite the data files from scratch, which will remove padding and store them on disk as efficiently as you are going to get. However you will need to have ~2x the size on disk to do so (actually less, but it's a decent guide).

One other thing to bear in mind here - repair and compact remove padding. The padding factor varies between 1 (no moves of documents caused by documents growing), to 2 (lots of moves caused by documents growing). Your padding factor of ~1.67 would indicate you are growing (and hence causing moves) quite a bit.

When you compact or repair a database you remove that padding - subsequent document growth is therefore going to trigger even more moves than before. Because moves are relatiely expensive operations, this can have a serious impact on your performance. More info here:

http://www.mongodb.org/display/DOCS/Padding+Factor

  • Thanks for your response @Adam, I am somewhat familiar with padding factors and compacting, what confuses me in this instance is that, no matter how effective compaction is we should never be able to store more data in the database than we're storing on harddisk! i.e., how do you fit 5.6GB of mongo data in 4.2GB of disk? – Chris W. May 22 '12 at 1:31
  • 4.2GB of disk is just the data, 5.6GB is the data plus indexes, and then for actual disk size you are probably going to have to look at the database level statistics instead – Adam C Jun 7 '12 at 16:46
  • I ran into the same thing! What is strange is that in their doc it says size does not account for indices: "Additionally size does not include the size of any indexes associated with the collection, which the totalIndexSize field reports." – MatijaSh May 11 '17 at 11:13
  • The reason may be that size displays uncompressed data size, while storage size takes compression into the account. It's described on db level here, but seems to be applicable for collection as well: docs.mongodb.com/manual/reference/command/dbStats/… – MatijaSh May 11 '17 at 11:46
1

For mongodb > 3.x

For MMAPv1: 
datasize < storageSize

but For wiredTiger
datasize > storageSize (most cases due to compression but may be
                        storageSize greater, it varies on condition like
                        compression technique, whether compact/repair 
                        command run or not)

For db.getCollection('name').stats()

size = total size in memory of all records in a collection + padding (excluded index size + record header which is 16 byte per header, header means  = field name)        
avgObjSize = avg size of obj + padding
storageSize =  total amount of storage allocated to this collection for document storage. (totalIndex size excluded)
totalIndexSize : totalIndexSize (compressed in case of wiredTiger)

For db.stats()

dataSize = document + padding
storageSize = document + padding + deleted space
fileSize = document + padding extents +  index extents + yet-unused space

We can delete unused space or hole by this

db.getCollection('name').runCommand( "compact" )

After running compact or repair command we can get exact storage size and data size difference.

Compression Technique in mongodb wiredTiger:

- snappy : good compression, low overhead
- zlib: better compression, more CPU
- none (we can disable compression, by default its enable in WT)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.