MongoDB scaling when working with files

Question

I have read through the MongoDB manual about sharding and replicasets.

However, I would like to know if the following can be achieved with sufficient performance (read/write):

• Saving 10,000,000 files via GridFS on MongoDB instances
• Total file size of about 2TB without indexes and journals
• 10,000 writes / day
• 10,000 reads / day
• Querying for a document does not need respond instantly, 2 seconds is still acceptable

Setup in mind:

• 1 Replicaset of 3 nodes
• Each node with 32GB Ram, 2TB SSD
• 1 Mongos instance
• 1 Config server

In the case the proposed setup would not suffice to obtain the goal of storing 10,000,000 documents and writing/reading 10,000 a day with fairly good performance, what would be the next step to take?

• Adding more RAM to each node
• Adding more disk capacity to each node
• Adding Another replica set with the same configuration
• Sharding

I would like to avoid sharding as much as possible since I consider it best practice to avoid sharding whenever possible. I feel this would only complicate the topology unnecessarily and this would most likely be overkill in this case

Some advice would be much welcome.

Cheers

Answer 1

My answer is : It depends. If you are accessing files by _id field, which is already indexed then you don't need to add more memory soon.

The _id field which is type of ObjectID is 12 byte in size. That means it can hold up to 2^(12*8) files. 3 byte is for machine ID which is a hash value and has a fixed vale on the machine can be subtracted which gives you approx 2^72 files. For the reference, 2^20 is 1,048,576.

In terms of the memory, the index on the _id field needs 10,000,000 x 12 byte = 114 MiBytes. To be honest, I don't now how much overhead there will be for an index which holds 10 millions value but I don't think that it will need more than 1 Gigabyte.

Now, if your _id field is not a type of ObjectID than do the math.

In the gridfs, filename value of the files collection is also indexed. If you are not accessing files using filename, then you may leave it blank and drop the index for the filename.

On the other side, if you will add some metadata to the files you added and want to query the files according to those metadata, then you should have indexes for those metadata and do the math again.

I have a production environment which has over 3,000,000 pdf files (takes 180 Gig space on the disk). My server is a virtual server which has 4 vCPU and 4 Gig RAM, still no problem. The specs you provided is way too high for your needs. You can save billions of files with those servers. Especially if you have SSD. Because even if your indexes do not fit into the memory, swapping will be very fast, you won't even notice a slowdown.

Answer 2

We've got ~16.200.000 files with total size of ~4.8 terabytes in mongo/gridfs.

25.000 - 30.000 files are added daily.

grid_fs database (reference gridfs spec) experiences about 50-300 queries per sec depending on the time of the day.

We have a pair of servers in replica set, both are single quad core cpu, 32 gigs of ram and a bunch of slow 2tb data disks in raid6.

Everything runs smoothly.