Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I need to create a database of time series, and perform the following tasks:

  • create new time series
  • update existing time series
  • query one or several time series at once (for instance all time series for the same date etc...)

Is Mongo adapted to that and if yes, how should I structure the database? (one time serie = one document? Or one document = one entry of the time serie, and all these documents form the collection which is the entire time series?)

I am a bit lost here and I find it difficult to find any information as usually Mongo is presented as very flexible so the user has the choice in the infrastructure.

Any link to tutorial that specifically explain how to manage time series in Mongo is very much welcome.

Thank you!

share|improve this question

2 Answers 2

up vote 3 down vote accepted

I suggest a single time series entry per document. There are some problems with storing multiple entries per document:

  • a single document is limited to a certain size (currently 16 MB); this limits how many entries can be stored in a single document
  • as more entries are added to a document, the entire document (and time series) will needlessly be deleted and reallocated to a larger piece of memory
  • queries on sub-documents are limited compared to queries on regular documents
  • documents with very flat structures (like one sub-document for each second) are not performant
  • the built-in map-reduce does not work as well on sub-documents

Also note a timestamp is built-in to the default MongoDB ObjectId. You can use this if the time series precision is less than one second.

Here is an example BSON document from an event logging library that uses MongoDB:

Example format of generated bson document:
{
    'thread': -1216977216,
    'level': 'ERROR',
    'timestamp': Timestamp(1290895671, 63),
    'message': 'test message',
    'fileName': '/var/projects/python/log4mongo-python/tests/test_mongo_handler.py',
    'lineNumber': 38,
    'method': 'test_emit_exception',
    'loggerName':  'testLogger',
    'exception': {
        'stackTrace': 'Traceback (most recent call last):
                       File "/var/projects/python/log4mongo-python/tests/test_mongo_handler.py", line 36, in test_emit_exception
                       raise Exception(\'exc1\')
                       Exception: exc1',
        'message': 'exc1',
        'code': 0
    }
}

Since an event log is similar to a time series, it may be worth studying the rest of the code. There are versions in Java, C#, PHP, and Python.

Here is another similar open source project: Zarkov


[update] In response to @RockScience's comment, I've adding some more references:

share|improve this answer
    
that's going to be a LOT of documents if my time series has intraday data for several years!!! is it not an issue to have so many documents? Coming from a sql background, I just find it not very memory effective. (As there will be a lot of repetition for all the data point of the same time series) –  RockScience Sep 11 '13 at 2:51
    
@RockScience: MongoDB, like many other NoSQL databases, eschew normalization and memory efficiency in favor of other things like flexibility, speed, and reduced CPU usage. If you need memory efficiency, MongoDB might not be the right solution for you. MongoDB copies the full text name of each field into every document, for crying out loud! Anyways, I've updated my answer with a few more resources, including a case study of how MongoDB was used to store a very large time series. –  Leftium Sep 11 '13 at 15:28
    
thanks for the aditional references –  RockScience Sep 12 '13 at 0:48

I found this question on SO (http://stackoverflow.com/questions/4814167/storing-time-series-data-relational-or-non) where the OP asks how to store a time series. Although his question is more-based around using a NoSQL database or a RDBMS, and you seem pretty set on using a NoSQL db.

Also found this article on "The Unique Database Requirements of Time-Series Data" that might be useful.

Hope this helps.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.