Keep files in Mongo as part of the data

Question

I want to create a news app that works in the following way:

The site will contain articles, and each article with have multiple versions. Each version is a docx file. The article itself does not have any content apart from the versions. The collection will look like this:

{
   Id:123,
   Versions: [{id:321, docx:"blob"},{id:444, docx:"blob"}]
}

I've read that Mongo allows you to save files in a format called gridFs, but from my understanding it keeps a different collection for the files and does not allow you to keep the files as a part of a custom collection.

Can someone suggest what is the best way to store the data in the way I described?

Answer 1

The problem is that MongoDB does not support file versions natively. So all we can do is to store the different versions and have some sort of metadata indicating the versions, then find the according file.

Adding the complete metadata to fs.files (the wrong way)

GridFS can store custom data in the metadata field of the files collection.

> db.fs.files.find().pretty()
{
    "_id" : ObjectId("571217193380e920f9b57e8a"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:42:34.109Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar"
}
> db.fs.files.update(
    {"_id" : ObjectId("571217193380e920f9b57e8a")},
    {$set:{metadata:{Versions: [{id:321, docx:"blob"},{id:444, docx:"blob"}]}}}
  )
WriteResult({ "nMatched" : 1, "nUpserted" : 0, "nModified" : 1 })
> db.fs.files.find().pretty()
{
    "_id" : ObjectId("571217193380e920f9b57e8a"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:42:34.109Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar",
    "metadata" : {
        "Versions" : [
            {
                "id" : 321,
                "docx" : "blob"
            },
            {
                "id" : 444,
                "docx" : "blob"
            }
        ]
    }
}

Of course, this would not make much sense, since you'd have the versions attached to a single file.

Use the uploadDatefield (the easy way)

Each time a file is saved into GridFS, MongoDB automatically adds the field uploadDate, which can be used to determine specific versions:

> db.fs.files.find().pretty()
{
    "_id" : ObjectId("571217193380e920f9b57e8a"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:42:34.109Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar"
}
{
    "_id" : ObjectId("57121ac13380e9212719bc5d"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:58:09.542Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar"
}

Note that for both files, the filename is the same, as is the md5 checksum.

Now, getting the latest version is pretty easy:

> db.fs.files.find({filename:"winstone.jar"}).sort({uploadDate:-1}).limit(1).pretty()
{
    "_id" : ObjectId("57121ac13380e9212719bc5d"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:58:09.542Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar"
}

Add the version to the fs.files document (the custom way)

When storing the file, you can add the version as an arbitrary field, which, as the documentation states, is perfectly ok:

Applications may create additional arbitrary fields.

> db.fs.files.find().pretty()
{
    "_id" : ObjectId("571217193380e920f9b57e8a"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:42:34.109Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar",
    "version" : 1
}
{
    "_id" : ObjectId("57121ac13380e9212719bc5d"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:58:09.542Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar",
    "version" : 2
}

Note the version field.

Getting the latest version of a file is as straightforward as it can be

> db.fs.files.find({filename:"winstone.jar"}).sort({version:-1}).limit(1).pretty()
{
    "_id" : ObjectId("57121ac13380e9212719bc5d"),
    "chunkSize" : 261120,
    "uploadDate" : ISODate("2016-04-16T10:58:09.542Z"),
    "length" : 358383,
    "md5" : "49ea7daa98666d87e5ca3208855eed0d",
    "filename" : "winstone.jar",
    "version" : 2
}

The general problem

But there is a serious drawback with all of these approaches: You do not save the diff in case a document is saved twice, you save the whole document again.

This leads me to the conclusion that MongoDB is not really suited for your use case, unless you can live with the fact that each version will be a full copy of the data.

Personally, I'd probably use Git Large File Storage to store the files and have a metadata collection. Without knowing a lot more about your use cases, it is hard to say if that suits you.