1

I'm trying to put a bunch of worldwide climate data into some kind of database structure so I can quickly access a collation of all 10 datasets simultaneously with fast indexing and hopefully efficient (space-wise) storage. It is important to be able to look up specific lat-long coordinates and quickly get all data recorded there, so my index is based on the location.

With the full range of data from 1901-2014, there are 360 (latitude) * 720 (longitude) * 12 (months) * 113 (years) * 10 (datasets) == 3,514,752,000 32-bit numerical readings, which can be found compressed here:

https://crudata.uea.ac.uk/cru/data/hrg/cru_ts_3.23/cruts.1506241137.v3.23/cld/

At first I tried making a document per month per location, with each document containing all 10 of the different datatypes (351,475,200 documents). This was quite fast to insert, but needed lots of disk space (something I'm trying to avoid). I hadn't yet discovered collection compression in MongoDB 3.x when I tried out this path, so I'm not sure to what extent that would help here.

The new approach I'm trying stores one document for each lat/long coordinate (360 * 720 documents total), each document containing 10 arrays that get upserted ($pushed) into. I imagine the constant query/write is slower than a plain write, though I'm using a unique index, and the lookup time should be effectively instantaneous.

This is much more space efficient, but running it now with a large dataset, I've noticed that the first few million points get inserted/upserted a lot quicker than those that come later, presumably because the collection size is growing and causing a lot more cache invalidations over time. At about 25% of the way through parsing the tsvs to MongoDB, I'm getting a write speed of about (only) 2000 documents per second. The entire operation will take about 40 hours to complete at this rate, but I fear that it will get increasingly (even) slower proportional to the db size.

I'm doing the number-crunching and sending the documents via node.js on my standard iMac (2016) with 8GB RAM, no fancy db servers involved (unfortunately).

Can anyone tell me whether relating to whether this is likely to slow down even more over time (which would necessitate a different approach to storing/inserting the data), or whether it's worth just waiting it out (I plan to do this insert operation once a year at most). I wonder whether mongoimport would be faster, not sure it works with $push though.

  • As feared, it gets increasingly slower the larger the database gets. At about 50% complete it slowed down to 1000 docs per second. Pretty much unusable. I've given up on the current implementation because the size storing everything in arrays is still too big anyway. I'd still be interested on any opinions on how to best store the data though – ephemer Mar 21 '16 at 16:00
  • Probably the best thing is to accept that you need a lot of disk space to store a lot of data, and design accordingly. – Vince Bowdren Feb 27 '17 at 16:26
  • @VinceBowdren in case you're interested, I ended up not using a database at all, instead saving the data in my own binary format (basically just as 16-bit integers + a header). This ended up about an entire order of magnitude smaller than the compressed (!) MongoDB database (1.7 GB vs 25+ GB), with sub-millisecond access times – ephemer Feb 28 '17 at 19:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.