1

Newbie here.

TL;DR : What is the best way to compact data in MongoDB? Will it use much more space than other database systems?

I wanted to create a database for something more or less like a forum. I started learning MySQL, but I don't need a RDBMS for what I want (most of my relations are 1 to 1 (at most, 1 to Many), even if I have a little data redundancy I'd rather store the information as field-and-value pairs of a document, I think it's faster for JSON than the many table joins in MySQL I would otherwise need). I also prefer JavaScript to SQL.

But looking now at the storage requirements, I find nothing like a TINYINT, CHAR(3), etc. I wanted to keep costs down with server's space, but at the same time record a lot of information (time, ratings, votes, comments, tags, karma, favorites, etc.). I hope I'm wrong, but MySQL seems to waste reasonably less space (although I imagine this is compensated in perfomance by MongoDB).

What are the best practices for reducing size in MongoDB? And am I completely wrong in thinking MongoDB uses more space than MySQL and other DBs (disregarding the benefits of joining tables in MySQL to avoid data redundancy)?

So far, these have been the best articles I found on it:
How to limit MongoDB database size (force size limits on the database)
https://www.compose.com/articles/sizing-and-trimming-your-mongodb/ (cap database and release prefetched space)
https://docs.mongodb.com/manual/reference/command/compact/ (use compact, specially on WiredTiger DBs)
https://stackoverflow.com/questions/2966687/reducing-mongodb-database-file-size (repairing data, and same as above)
As @Fyodor Glebov reminds in his answer, there are also 2 compression algorithms: snappy and zlib.

Thanks for any advices!

0

Because data is not normalised the storage requirements generally bigger.

Don't forget about compression. When you use gzip it will use more CPU power, but disk size is much smaller than snappy. It depends on your data.

Some MongoDB consultant made benchmarks about different compression algorithms, but this file not public. Maybe some MongoDB guy provide an improved answer ;)

  • No compression
  • Snappy (enabled by default) – very good compression, efficient use of resources
  • zlib (similar to gzip) – excellent compression, but more resource intensive

see the startup parameter storage.wiredTiger.collectionConfig.blockCompressor

Default: snappy

New in version 3.0.0.

The default type of compression to use to compress collection data. You can override this on a per-collection basis when creating collections.

Available compressors are:

none snappy zlib storage.wiredTiger.collectionConfig.blockCompressor affects all collections created. If you change the value of storage.wiredTiger.collectionConfig.blockCompressor on an existing MongoDB deployment, all new collections will use the specified compressor. Existing collections will continue to use the compressor specified when they were created, or the default compressor at that time.

  • I'm not sure which compression benchmark you're referring to, but some comparisons are included on New Compression Options in MongoDB 3.0 (official MongoDB blog) and Wired Tiger - how to reduce your MongoDB hosting costs 10x (a MongoDB Consulting Engineer's blog). – Stennie Jan 16 '17 at 19:19
  • Thanks a lot for your answer! And to @Stennie 's references! You're right, I'm slowly getting used to the idea that MongoDB's priority isn't disk space (inevitable in denormalization) and that disk space today is rather cheap. I got scared about mentions of it being a resource hog, but it does seem tameable. – flen Jan 19 '17 at 11:07
  • @flen Aside from the compact command which links to the current MongoDB manual, the articles you referenced in your description describe the older MMAP storage engine. In MongoDB 3.2+ the default storage engine is WiredTiger, which has significantly different behaviour include data and index compression. Irrespective of storage engine, the underlying data stored in MongoDB has always had types (see BSON spec) so you can choose more appropriate representations. For example, see: How should I store boolean values?. – Stennie Jan 19 '17 at 11:10
  • Thank you @Stennie , I noticed the change to WiredTiger but it's a good thing to emphasize it. I know that BSON has different data types, but that's where it hurts... while I can use TINYINT, (VAR)CHAR and ENUM in MySQL as ways to keep data small, in MongoDB ObjectID (for example) takes 12 bytes! Integers in a 64-bit server are always 8 bytes and small strings are also bigger because BSON records length etc. in it. I read somewhere that one can use "false", "true", "max and min keys" as 1 byte values for 4 options (e.g., white, black, light-gray, dark-grey). But this is a bad fix... – flen Jan 19 '17 at 13:20
  • My point was that there are different data types in BSON. For example, numbers can be 32-bit int, 64-bit int, 64-bit float, or decimal (which range from 4 to 16 bytes). Data storage is different as compared to fixed fields in relational tables (eg. compression & schema-on-read) but the schema generally supports use case efficiency rather than storage efficiency. However, the false/true/min/max example you found sounds like misuse: 4 named boolean fields would be saner, or you could use bitwise updates/queries for feature flags in a single int32 field if storage overhead is a key concern. – Stennie Jan 21 '17 at 20:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.