I am using MongoDB. I have a hardware device which continuously submits data to the database every 10 seconds for 4 hours a day. The data includes latitude, longitude, and other basic information of the device. I have indexed latitude and longitude for faster querying. I have used an event based document creation approach. That is, every time the device submits data, I am creating a document in my collection. The average number of documents created for per day will be 1,440. My document size is 300 bytes. Since I have 3 indices (including _id) the RAM consumption per document is going to be 8176*3 = ~24KB. I will have a million devices submitting data in a similar fashion. If this continues I will end up wanting 33,000 GB of RAM.

Am I assuming something wrong? What is the right approach to manage the documents in this case?

  • "The average number of documents created for per day will be 1,440" Is that total or just per device and we will get 1,440 events (documents) x 1,000,000 devices? – Ali Razeghi Jul 9 '15 at 0:44

If your base document with all the data is 300 bytes, and worst case scenario for storage usage scenario, your indexes are all 300 bytes each but just sorted differently, would get you 900 bytes of memory used per document for indexes and 300 bytes for the base.

1440 events * 1200 bytes (Base document + 3 indexes each at 300 bytes) = 1728000 bytes

1296000 bytes / 1024 = 1687.5 kilobytes

1265.625 kb / 1024 = 1.64 megabytes

So if you are firing off 1440 events per day per device, each single device will take up 1.64 mb of memory plus a bit more for metadata overhead. If you are firing off that many events for 1,000,000 devices it will be:

1647949.21875 mb / 1024 = 1609.32 gb

so roughly a total of 1.8 TB of ram will be needed to fully fit this into memory without utilizing any compression.

You can deal with the RAM issue by scaling out in shards or finding a few very expensive systems that can handle that much RAM (note I haven't heard of one and I'm not sure how many CPU's you'd need, this isn't realistic). You can shard but then you'd typically want 3 hosts per shard and it comes with it's own operational overhead. It also eats up a lot of power an data center space.

You can compromise and just deal with the fact that you will read from disk, if your app is OK with that. Here are some other solutions.

There are all flash SANs out there that measure returns in microseconds, with RAM delivering data in nanoseconds. It will be an order of magnitude slower than RAM but it is still insanely fast and might be just fine for your reqs. The problem with these is you need top of the line Fiber Channel HBAs to have the throughput.

There are FusionIO cards that could work very well. These will give you a PCI-E bus to connect to so throughput is no problem and you don't need expensive HBAs. They aren't cheap though and if your app is very time sensitive it still isn't good enough. FusionIO will typically give you a loaner card to test and they always make me drool. They have a marketing page on it here.

  • When I checked in Robomongo, each index size was 8176. So for 3 indeices, we will have 3*8176=23KB of RAM usage/document. How is it only 300 bytes in the worst case? – Jaseem Abbas Jul 9 '15 at 5:25
  • What are you storing in your index that is bigger than 27x bigger than your base document? Your base document is everything. An index will typically have a subset of the fields, or all the fields but sorted differently so typically worst case scenario it will be equal size. I don't know of any 'computed index columns' in mongo. Maybe you're looking at all document indexes. Show us a sample output of your base document and indexing. Do your indexes contain each data field from the base document 27 times or is there something odd that is going on? – Ali Razeghi Jul 9 '15 at 19:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.