I'm trying to put a bunch of worldwide climate data into some kind of database structure so I can quickly access a collation of all 10 datasets simultaneously with fast indexing and hopefully efficient (space-wise) storage. It is important to be able to look up specific lat-long coordinates and quickly get all data recorded there, so my index is based on the location.
With the full range of data from 1901-2014, there are
360 (latitude) * 720 (longitude) * 12 (months) * 113 (years) * 10 (datasets) == 3,514,752,000 32-bit numerical readings, which can be found compressed here:
At first I tried making a document per month per location, with each document containing all 10 of the different datatypes (
351,475,200 documents). This was quite fast to insert, but needed lots of disk space (something I'm trying to avoid). I hadn't yet discovered collection compression in MongoDB 3.x when I tried out this path, so I'm not sure to what extent that would help here.
The new approach I'm trying stores one document for each lat/long coordinate (360 * 720 documents total), each document containing 10 arrays that get upserted (
$pushed) into. I imagine the constant query/write is slower than a plain write, though I'm using a unique index, and the lookup time should be effectively instantaneous.
This is much more space efficient, but running it now with a large dataset, I've noticed that the first few million points get inserted/upserted a lot quicker than those that come later, presumably because the collection size is growing and causing a lot more cache invalidations over time. At about 25% of the way through parsing the tsvs to MongoDB, I'm getting a write speed of about (only) 2000 documents per second. The entire operation will take about 40 hours to complete at this rate, but I fear that it will get increasingly (even) slower proportional to the db size.
I'm doing the number-crunching and sending the documents via node.js on my standard iMac (2016) with 8GB RAM, no fancy db servers involved (unfortunately).
Can anyone tell me whether relating to whether this is likely to slow down even more over time (which would necessitate a different approach to storing/inserting the data), or whether it's worth just waiting it out (I plan to do this insert operation once a year at most). I wonder whether mongoimport would be faster, not sure it works with