What is the recommended (type) of database design for storing large amounts of time-series data?

The dataset:

  • 2 million time-series;
  • Each of the time-series contains around 500 timestamps;
  • At each of the time-stamp there are multiple variables (meta-data);
  • It would be optimal if meta-data could later easily be ingested into the database (preferably with a Python API) .

I have looked into PostgreSQL, but so far I have been struggling to set it up for many time-series.

| improve this question | | | | |
  • 1
    describe the meta-data that goes with each timestamp how complex is this data? – zsheep Feb 26 at 20:36
  • you probably want a database that is optimized for time-series analysis. Any database can do it, but some query patterns are specific to time-series that some databases aren't optimized for. timescale.com is based on postgresql but is focused on time-series analysis – Neil McGuigan Feb 26 at 23:05
  • The meta-data will features to enrich the dataset. They might be scraped or coming from other datasets. It is for a natural sciences project, so things like humidity etc. I don't expect them to have complex nested structures. – sir_olf Feb 27 at 9:19
  • Thank you for suggesting Timescale. I remember having a look at that earlier. Probably I will have to set up the structure as a hypertable. E.g, blog.timescale.com/tutorials/… – sir_olf Feb 27 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.