-1

NOTE: I am using geographic locations here in my question, but those who do image processing likely face similar questions/issues. That is why I've titled the question using unstructured point could data for general applications.

I have a dataset representing geographic locations in 3D space (i.e. x,y,z) which contains about 150,000,000 points. The data was provided to me in the form of a geoTiff, whose filesize is 554.8 MB. When I load the data into Matlab (could be any program) for post-processing, it takes about 5 seconds to upload and utilizes about 1.3GB of memory.

Once the data is uploaded, I'm only actually working with about 75,000 points, which is about 0.05% of the entire dataset. Therefore I wanted to explore a more efficient way of working with the data, where in the most general case, the geographic locations would be unstructured. That being said, I thought it would be worth looking into storing the dataset in a NoSQL database like MongoDB, then using geo-spatial queries to read in the 75,000 points without having to load the entire dataset into memory.

After populating a Mongo collection, the collection stats show "storageSize" : 5182734336.0 and "totalIndexSize" : 3285737472.0. This obviously occupies way more than 554.8 MB of disk space for the geoTiff file. At first this was rather disappointing, but then I realized that the geoTiff file I was working with only stores the z data since the point cloud is on a structured grid. Therefore, I should take 5GB / 3 = 1.6GB to make a fair comparison (ignoring the "totalIndexSize"). Considering I stored the data in Mongo using geoJSON format, which contains additional info/text along with the values, I'd say 1.6GB seems reasonable in comparison to the 554.8 MB of the geoTiff. So as far as disk storage requirements, I think for the general case of unstructured point cloud data, Mongo seems like a reasonable solution.

All was good until I noticed the amount of RAM used by the mongod process after populating the collection. Looking at my system monitor, the mongod process is using over 7GB of RAM (which is nearly half of what I have available). Even if the queries are faster than loading the geoTiff, the amount of RAM used limits my resources (7GB Mongo vs 1.3GB upload geoTiff in Matlab). In this particular case, it seems it is only good to use a database when the data is read every 5 seconds or less. Otherwise, it's faster to read the data in from the geoTiff file when needed (5 seconds per load) and save the RAM.

I'm pretty sure the amount of RAM required is tied to the ability to make quick queries, but on my machine, the RAM is a big sacrifice when performing other processes. Yes, I know, a better fit would be to use a database server, but I'm just exploring potentials here.

This is my first attempt at using MongoDB, so maybe I'm not using it right. If I want to access the data through Matlab (using the Java driver), do I have to keep the mongod process running on my machine for the data to be available or can the driver start the mongod process then shut it down after data extraction? Would this require a large "startup" time? Can I "startup" a mongod process targeting a specific collection or does the mongod process load all collections into RAM? It doesn't seem logical to have the RAM requirements on the same order as disk requirements? For example, couldn't I make 1 collection visible on RAM then have 10 others "sleeping" on disk?

Above all, I'm wondering how people address unstructured point cloud data in an efficient way? Is the filesystem approach better than the database approach? Can I somehow make the best of both worlds? The case I presented here is only representative of 1 dataset and I actually have 50 more that need processing/storage. I'm wondering if a database approach is even worth digging more into?

1

RAM utilization

MongoDB is a DBMS highly optimized for being run on servers. I even have a mantra: "Always run MongoDB on a dedicated server." With no processes competing for ressources, that is. The reason is that MongoDB makes heavy use of about all system ressources, most notably (pun intended) RAM. It will use up to 85% of your physical memory to keep the indices (which you correctly deduced is the stuff that makes your queries fast) and the working set (LRU-evicted "cache" for data) in RAM. While there are ways to limit MongoDBs RAM utilization, here is my advice with regards to that: simply Don't.

Running MongoDB on a remote machine

I have not worked with MatLab, but as long as you can enter some sort of connection configuration (and everything else would be a little strange, to say the least), I do not see why it would not be an excellent idea to run MongoDB on a cloud instance. Fire one up, load your data, do your processing, destroy the instance. Rinse, repeat. Should not cost you more than a couple of bucks. Be it on an IaaS (AWS, Azure) or SaaS (MongoDB's own Atlas, mlab) platform. If you use the former, just make sure the image you use is properly configured. MongoDB has extensive documentation with regards to that.

Is it worth running geospatial queries in a DBMS

There are plenty of implementations of geospatial queries for a rather wide variety of DBMS. Maintained, aka people putting effort into developing a feature. Go figure. ;)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.