0

I have to save a lot of documents that I'll show on a web and I have a Mongo database to save them.

I know that Mongo can autogenerate a random ID for each element in a collection and I also know that you can create a hashed index then to increase the speed search.

I'll use the document SHA1 hash (or maybe other) as identifier in the web to each document, and I was wondering if it could be a good idea to use it as custom index on Mongo.

It doesn't matter if someone makes the SHA1 to a random document to search it on the web, I mean, the security it's not a problem because the documents will be public.

I saw that other people have used custom index and they say that it's useful

In my opinion, doing it would have these advantages:

  • It's a faster way to find documents, because you can search by the documet hash and not by whole document.
  • The ID it's already an index so you will save one index.
  • You may save some storage because you don't need to save the hash separately.
  • The SHA1 size is smaller than default Mongo id, so you will be able to have more index in memory at same time.

And as disadvantages:

  • You have to manage the collisions manually. (I may use a "bigger" hash function)
  • You need to generate the hash by yourself.
  • it will be harder to short the documents and you may have extra keys.

But there are some disadvantages?

2

I know that Mongo can autogenerate a random ID for each element in a collection and I also know that you can create a hashed index then to increase the speed search.

Hashed indexes in MongoDB are typically used to create a suitable shard key for data distribution where the original field value is monotonically increasing (for example, default ObjectIDs). The underlying index format is still a B-Tree, so a hashed index does not change the speed or complexity to look up an index value.

I'll use the document SHA1 hash (or maybe other) as identifier in the web to each document, and I was wondering if it could be a good idea to use it as custom index

ObjectIDs are pseudorandom identifiers designed to be independently generated and reasonably unique with roughly ascending order (based on a leading timestamp component); SHA1 hashes require unique input values and are unordered. ObjectIDs are generated by clients/drivers by default.

Using a custom _id format is fine, but you should consider desired attributes like uniqueness (required for a primary key) as well as ordering (depends on your use case) and independent generation (required for effective scaling of a distributed deployment).

It's a faster way to find documents, because you can search by the ; hash and not by whole document.

As mentioned above, this is not a faster search solution. If you already know the primary key (_id value), looking it up as a hash value or as an ObjectID is equivalent.

The ID it's already an index so you will save one index.

True.

You may save some storage because you don't need to save the hash separately.

_id is already a unique reference, so you wouldn't necessarily need another unique value.

The SHA1 size is smaller than default Mongo id, so you will be able to have more index in memory at same time.

A normal SHA-1 hash is 20 bytes, which is 8 bytes more than an ObjectID. Perhaps you are thinking of the truncated SHA-1 hashes used by git (typically around 7 bytes). I'm not sure what the relative probability of collisions would be, but smaller key spaces increase the possibility.

There are also factors to consider other than the size of the _id value. Indexes in MongoDB use prefix compression by default, which is helpful when you have a values like ObjectIDs which start with 4 byte timestamps. Indexes with random values may end up creating larger data structures.

For an example comparison of index prefix compression, this older blog post from John Page might be useful: How to reduce your MongoDB hosting costs.

The best way to test the impact of data model changes would be to generate some representative test data for your planned approach. There are mock data tools that can help with creation of plausible data (for example, mgeneratejs).

Not the answer you're looking for? Browse other questions tagged or ask your own question.