I have 2TB of data currently spread across many CSV files that looks like:

 'rec_date': "2010-01-29", 
 'rec_time': "09:15:00", 
 'site_no': '46',
 'data_owner': '1',
 'flow_period': '5',
 'vehicle_count': '60',
 'detector': '8'

I would like to insert this into mongodb so that I can quickly do queries that typically get the vehicle volume for a site and its surrounding sites for the past n Mondays of the past m years. (I wish to analyse the history of a site to determine if there's abnormal traffic flow for this time of day at this time of year).

Initially I wanted to store a document for each site, that contained its entire history, like so:

    site_no: 46,
    history: [{
            $date: 1264756500000,
            readings: [{sensor:1,vehicle_count:60},
            $date: 1264756800000,
            readings: [....]

But this would make each document very large, as the entire history for each site will be more than 16MB. More traffic volume data is coming in every 5 minutes from every site and I would need to be appending to the history array very often.

So my question is, how should I format my data so that I can perform the queries I want but minimising data redundancy? Should I just make a new record for every sensor reading?


Yes, create a new record for every reading.

MongoDB and other document stores are designed for efficient retrieval of de-normalized or irregularly structured data. A MongoDB collection is essentially a set of key-value pairs with a rich value type. Updating the value is an atomic operation, which means that constantly appending to a list within the value is not going to be efficient and, as you mentioned, you will eventually hit the document size limit.

| improve this answer | |
  • Rapid aggregation over dense dimensions (e.g. traffic by site and day-of-week) is exactly the kind of query that MOLAP systems and column-oriented databases are great at, but also straightforward for traditional RDBMSs. What drove the choice to use MongoDB? – user2313838 Jan 20 '15 at 16:59
  • Well for one, I'd just like to try a new technology; I also heard it handles very large volumes of data like I have very well. – Jonno_FTW Jan 21 '15 at 2:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.