NOTE: I am using geographic locations here in my question, but those who do image processing likely face similar questions/issues. That is why I've titled the question using
unstructured point could data for general applications.
I have a dataset representing geographic locations in 3D space (i.e. x,y,z) which contains about 150,000,000 points. The data was provided to me in the form of a geoTiff, whose filesize is 554.8 MB. When I load the data into Matlab (could be any program) for post-processing, it takes about 5 seconds to upload and utilizes about 1.3GB of memory.
Once the data is uploaded, I'm only actually working with about 75,000 points, which is about 0.05% of the entire dataset. Therefore I wanted to explore a more efficient way of working with the data, where in the most general case, the geographic locations would be unstructured. That being said, I thought it would be worth looking into storing the dataset in a NoSQL database like MongoDB, then using geo-spatial queries to read in the 75,000 points without having to load the entire dataset into memory.
After populating a Mongo collection, the collection stats show
"storageSize" : 5182734336.0 and
"totalIndexSize" : 3285737472.0. This obviously occupies way more than 554.8 MB of disk space for the geoTiff file. At first this was rather disappointing, but then I realized that the geoTiff file I was working with only stores the
z data since the point cloud is on a structured grid. Therefore, I should take 5GB / 3 = 1.6GB to make a fair comparison (ignoring the "totalIndexSize"). Considering I stored the data in Mongo using geoJSON format, which contains additional info/text along with the values, I'd say 1.6GB seems reasonable in comparison to the 554.8 MB of the geoTiff. So as far as disk storage requirements, I think for the general case of unstructured point cloud data, Mongo seems like a reasonable solution.
All was good until I noticed the amount of RAM used by the
mongod process after populating the collection. Looking at my
system monitor, the
mongod process is using over 7GB of RAM (which is nearly half of what I have available). Even if the queries are faster than loading the geoTiff, the amount of RAM used limits my resources (7GB Mongo vs 1.3GB upload geoTiff in Matlab). In this particular case, it seems it is only good to use a database when the data is read every 5 seconds or less. Otherwise, it's faster to read the data in from the geoTiff file when needed (5 seconds per load) and save the RAM.
I'm pretty sure the amount of RAM required is tied to the ability to make quick queries, but on my machine, the RAM is a big sacrifice when performing other processes. Yes, I know, a better fit would be to use a database server, but I'm just exploring potentials here.
This is my first attempt at using MongoDB, so maybe I'm not using it right. If I want to access the data through Matlab (using the Java driver), do I have to keep the
mongod process running on my machine for the data to be available or can the driver start the
mongod process then shut it down after data extraction? Would this require a large "startup" time? Can I "startup" a
mongod process targeting a specific collection or does the
mongod process load all collections into RAM? It doesn't seem logical to have the RAM requirements on the same order as disk requirements? For example, couldn't I make 1 collection visible on RAM then have 10 others "sleeping" on disk?
Above all, I'm wondering how people address unstructured point cloud data in an efficient way? Is the filesystem approach better than the database approach? Can I somehow make the best of both worlds? The case I presented here is only representative of 1 dataset and I actually have 50 more that need processing/storage. I'm wondering if a database approach is even worth digging more into?