2

I really like using mongodb to store my data and recently I tried out GridFS and it really fits my use case.

My problem with it is the space requirement, which seems quite odd. I have ~107GB of images in Amazon S3, which is around 1 million files (all images, mostly small ones). I made a simple Java project to download the images from S3 and insert them into two separate MongoDB GridFS collections (single server, 3.6.5, 64 bit, Windows Server 2016). The problem is, when the upload/download completes, the GridFS collections take more than 300GB storage on the server. Is this acceptable for this kind of collection or should I worry about the tripled size?

Note: I simply inserted the images using the Java Mongo Driver (Spring Boot) without any significant change, the problem is with the image chunks. I do not delete or update any images (I defined a unique index for the MD5 field though, to ignore image duplication), thus compact and repair does not change the collection sizes. As much as I can see, the collections are not overly preallocated (I don't think my problem is similar to this: Huge size on mongodb's gridfs. Should I compact? )

Also, currently it is a single mongodb server, without a replica set.

Thank you very much for your help!

  • 1
    It's basically expected. GridFS breaks the data into "chunks" and stores those binary parts within documents just like anything else in MongoDB. Those documents have some additional overhead by definition and additionally in terms of storage they get allocated a pre-set amount, being typically larger than they would be expected to grow. There are things that can be done to "tune" that allocated amount, as well as enabling compression in storage which is another option. – Neil Lunn Jun 6 '18 at 7:34
  • 1
    Also, whilst you may not be at "that other problem yet", it is basically inherent to storing data within the database. If you don't have a compelling reason why you might possibly need "distributed data" access to stored files, and are just simply looking for "cheap options", then anything essentially filesystem based is usually the better option. Using MongoDB for storage does have overheads. You either want those for a specific reason, or you're looking in the wrong place. – Neil Lunn Jun 6 '18 at 7:37
  • @ NeilLunn,Even in MongoDB the GridFS is using to store the BSON document size, who has larger than 16 MB. – Md Haidar Ali Khan Jun 6 '18 at 7:51
  • How did you determine that the collection is taking >300GB in GridFS and 107GB in S3? If you used ls -l it will be helpful to include those output in the question. Please also post the output of db.fs.chunks.stats() (assuming your chunks collection is using the default name of fs.chunks) – kevinadi Jun 14 '18 at 2:59
  • @KevinAdistambha S3 gives you the size of a bucket plain and simple, same with mongoDb. I migrated everything and dropped gridfs, so I can't give you more details now. – appl3r Jun 14 '18 at 7:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.