How should I store time series in mongodb

Question

I need to create a database of time series, and perform the following tasks:

• create new time series
• update existing time series
• query one or several time series at once (for instance all time series for the same date etc...)

Is Mongo adapted to that and if yes, how should I structure the database? (one time serie = one document? Or one document = one entry of the time serie, and all these documents form the collection which is the entire time series?)

I am a bit lost here and I find it difficult to find any information as usually Mongo is presented as very flexible so the user has the choice in the infrastructure.

Any link to tutorial that specifically explain how to manage time series in Mongo is very much welcome.

Thank you!

Answer 1

I suggest a single time series entry per document. There are some problems with storing multiple entries per document:

• a single document is limited to a certain size (currently 16 MB); this limits how many entries can be stored in a single document
• as more entries are added to a document, the entire document (and time series) will needlessly be deleted and reallocated to a larger piece of memory
• queries on sub-documents are limited compared to queries on regular documents
• documents with very flat structures (like one sub-document for each second) are not performant
• the built-in map-reduce does not work as well on sub-documents

Also note a timestamp is built-in to the default MongoDB ObjectId. You can use this if the time series precision is less than one second.

Here is an example BSON document from an event logging library that uses MongoDB:

Example format of generated bson document:
{
    'thread': -1216977216,
    'level': 'ERROR',
    'timestamp': Timestamp(1290895671, 63),
    'message': 'test message',
    'fileName': '/var/projects/python/log4mongo-python/tests/test_mongo_handler.py',
    'lineNumber': 38,
    'method': 'test_emit_exception',
    'loggerName':  'testLogger',
    'exception': {
        'stackTrace': 'Traceback (most recent call last):
                       File "/var/projects/python/log4mongo-python/tests/test_mongo_handler.py", line 36, in test_emit_exception
                       raise Exception(\'exc1\')
                       Exception: exc1',
        'message': 'exc1',
        'code': 0
    }
}

Since an event log is similar to a time series, it may be worth studying the rest of the code. There are versions in Java, C#, PHP, and Python.

Here is another similar open source project: Zarkov

---

[update] In response to @RockScience's comment, I've adding some more references:

• Very verbose case study of how MongoDB was used to store time series. Interesting parts:
  • Slide 25: how much data? 20 4-byte data points @ 1 minute intervals for 10 years (not apparent if these are actual statistics or just projected estimates)
  • Slide 66: shows sample MongoDB document used to store time series data
• Best practices for storing log data, straight from the creators of MongoDB

Answer 2

I found this question on SO (https://stackoverflow.com/questions/4814167/storing-time-series-data-relational-or-non) where the OP asks how to store a time series. Although his question is more-based around using a NoSQL database or a RDBMS, and you seem pretty set on using a NoSQL db.

Also found this article on "The Unique Database Requirements of Time-Series Data" that might be useful.

Hope this helps.

Answer 3

Yes definitely, NoSQL database better suits storing timeseries data than traditional RDBMS.

Yes MongoDB is exceptionally adapted to this use case.

-How should you structure the database? One document = one time series input VS multiple time series.

The answer is to store in one document multiple timeseries. Having less documents will help the performance with less reads. One trick is to prepare your document with the predefined values. This will optimize updating the document by avoiding Record Padding.

Here's a schema example on how to optimally store an hour worth of timeseries with a minute interval:

{
  timestamp_hour: ISODate("2015-07-02T23:00:00.000Z"),
  type: “memory_used”,
  values: {
    0: 999999,
    1: 1000000, 
    …,
    58: 0,
    59: 0
  }
}

You initiate it with 0 values, and then updates will be optimized. The reads are optimized because one document is read instead of 60. If you need to store a day worth of data, or a month you proceed with the same technique, you get the idea.

Here's the link to a tutorial that specifically explains how to manage time series in MongoDb from the official MongoDb Blog: http://blog.mongodb.org/post/65517193370/schema-design-for-time-series-data-in-mongodb