1

I am building a website that allows users to retrieve a bunch of open source chart (eg UN data). I don't need to be able to query the data itself. I will instead extract certain meta data (keywords, date ranges, etc) that will be stored separately in a structured format for querying.

Since I don't need to query the data itself, would a NoSQL database offer any advantage over something like a Postgres JSONB field? I'm primarily thinking about read performance, database size, and cost (I realize cost will depend on a number of factors, so maybe that's not easy to anticipate at this high level).

  • speed will probably be about the same either way. use json instead of jsonb if you don't need to query it. – Jasen Feb 3 at 1:37
  • @Jasen why is that? simply for faster write performance, or are there other advantages of json over jsonb? – maxedison Feb 5 at 1:33
  • 'json' allows things that 'jsonb' does not, (things not supported by javascript but permitted by RFC7159). Also json (being essentially text) preserves key order, jsonb does not. – Jasen Feb 5 at 8:53
2

Neither. What you need is a file system, or cloud blob storage. Choose one with high compression ratios.

Typically data is used intensely for a short while then touched only infrequently thereafter. Choosing a storage service with tiers will allow further cost saving.

As the service becomes popular it is likely several users will have the same dataset. Storage service de-duplication could allow further cost reduction.

Store the meta data in whichever flavour of DBMS you choose for the main application. This will reduce the overall complexity and utilise existing expertise. Relational would be my preference but document storage would work, too.

MongoDB adds empty space to the written data so the on-disk length follows a power-of-two rule. This could waste a lot of space depending on how your source JSON works out. This is a lot like the fill-factor in RDBMS B-Tree indexes. This behaviour can be removed in more recent versions.

Performance-wise I would think a solution based on the file system directly would be marginally faster than one involving a DBMS of either flavour. Both approaches will use the same underlying hardware - spindles, NICs, buses etc. The DBMS, no matter how efficient, is an additional layer in the stack between the disk and the client. If it is not adding additional value, through joins, filters, aggregates, security or whatever, I can't see how having it in the run-time path helps. The meta-data is a different beast to the JSON, of course, and will benefit from all the additional functionality a DBMS provides.

There's a trade-off to be had with compressed data. On one side the storage IO and network communication is faster when the data is compressed. Reading 10GB uncompressed data from disk and moving it over the network will always be slower than reading and moving the same data compressed to 7GB. On the other side is the CPU cost to decompress before usage. My experience from data warehouse environments, with gigabyte-scale reads and ten-plus cores, is that compression lowered elapsed time. In OLTP-like environments (small, random reads with frequent updates) compression is definitely not recommended. We don't have hardware specs, dataset sizes, usage patterns, pricing information or a measure of your speed-versus-cost tolerance so can't calculate this on your behalf. Likely testing at production scale will be the only way to answer this properly.

  • Users won't be allowed to upload their own datasets -- they will have read permissions only. I'll be the only one who can add new data. And I want to build the application with the assumption that datasets will be getting accessed frequently and therefore read performance is important. My sense is that the solution you're recommending sacrifices read performance for space/cost savings, correct? If so, I'd be interested to hear a recommendation that prioritizes read performance. (I'd also be interested to learn more about how MongoDB adds empty space) – maxedison Feb 5 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.