0

Context:

I am developing an application that requires handling tables that do not necessarily have an uniform structure (eg. different/extra columns). Those unstructured files also need to be grouped (say 5 table together makes a group, each table has a different content). And each file in the group needs to be versioned preferably in a simple and efficient way (eg. no need to copy the entire file if only a row is changed).

Current solution I thought of:

  • Define a schema with some standard fields that application needs, such as name of a file group, ..., timestamps, path_to_repository, hash_of_repository_content (to invalidate), status (eg. being_updated, for concurrency) }
  • At path_to_repository, I'll have a git repository and keep files in csv format (create a submodule for each file so that each file in a file group can be versioned independently).
  • In order to access those files, I'll get the repository and head will be the current version of the files. Then I can do the versioning easily, can keep tables of unlimited size without any structure and wouldnt be limited by mongodb's max file size. If there are any further information to be versioned I can simply add them to the repository.
  • To handle concurrency, I'll update a flag in the mongodb, so it would take care of the concurrency for me. If two applications simultaneously try to write into repository, they'd set a flag/lock first. if flag is already updated by someone else, they need to wait.

Question:

I am not sure is this is a good practice (whether I'm re-inventing the wheel or not, would it have a good performance, can this scale, if there is any problems I dont see). Any advice/experience/criticism would be great.

Probably irrelevant extra information:

this is going to be used by nodejs/mongoose and also by a python script.

1

The decision about what to put in the document is pretty much determined by how the data is used by the application.

The data that is used together as users documents is a good candidate to be pre-joined or embedded.

One of the limitations of this approach is the size of the document. It should be a maximum of 16 MB.

Another approach is to split data between multiple collections.

One of the limitations of this approach is that there is no constraint in MongoDB, so there are no foreign key constraints as well.

The database does not guarantee consistency of the data. Is it up to you as a programmer to take care that your data has no orphans.

Data from multiple collections could be joined by applying the lookup operator. But, a collection is a separate file on disk, so seeking on multiple collections means seeking from multiple files, and that is, as you are probably guessing, slow.

Generally speaking, embedded data is the preferable approach.

  • Thanks a lot for your answer. At the moment combined data size seem to be less than 16 MB, but I dont want to rely on this much as different users might have different needs/inputs. The usage is simply reading each document and using them as an input for an algorithm (this is done by a separate worker host, not by the server or db). The algorithm and the data are going to change very often and at any point users might decide to revert back to any state in history. That is why version management is quite important. – ozgeneral Jun 10 at 21:05
  • Reason that led me to consider using git and a file system is because there is no reliable size range and I'll need a good version management (preferably an established one that uses diffs and takes care of ids, comments, timestamps etc for me). And if I'll have to implement a workaround that cleans after itself in any case (referring to not leaving orphans behind etc), I might just as well implement this one. But my concern is if there is any possible performance/scalability issue that I might have overlooked or if there is already a nice tool that is built in that I'm not aware of – ozgeneral Jun 10 at 21:10
  • note: updated question to my current plan – ozgeneral Jun 10 at 21:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.