0

I have a quite big PostgreSQL database (with timescale plugin). Right now it is consuming about 500Gb on a SSD. Most of the data is in the form of time series. In most cases data older than a few months isn't really interesting.

My idea was to move that data to a cheap SATA hard drive instead of buying more expensive SSD's. Is that a good idea, and is there some good practice for implementing?

My naive implementation would be:
Keep two databases (or create a tablespace on the cheap HDD). Fetch every few hours data from "fast" (SSD) database to the "slow" database (HDD). Every few days, delete data from the slow database. Is this a good idea? I am happy to hear some feedback and better suggestions.

2

Here is a better architecture:

  • Create a new tablespace on the slow drive.

  • Set the storage parameters seq_page_cost and random_page_cost higher on that new tablespace so that the PostgreSQL optimizer knows that the disks are slower.

  • Partition the big time series tables by time ranges (use the same boundaries for all affected tables) so that you end up with a couple of dozen partitions for each.

  • Move the old partitions to the slow tablespace.

Then you still have all the data accessible.

Use PostgreSQL v11 or better for partitioning.

  • Thanks a lot for the fast answer. Does that mean that I would have to do the Partition all the time? Do you know is it possible to move the timescaledb partitions that way. As I understand the architecture, timescale already handles the partition part. That would be optimal. Then I could just move the old partitions to the slow and cheap HDD. Actually perfect solution - if that is possible. – MichaelRazum Jul 18 at 22:22
  • I don't know the details of timescaledb, but normally you can simply move partitions with ALTER TABLE table_partition SET TABLESPACE newtbsp;. – Laurenz Albe Jul 18 at 23:08
  • Thanks a lot will try that out :)! – MichaelRazum Jul 18 at 23:08
0

PM from Timescale here. Our next release is scheduled to introduce compression, so that will help with things. Tablespaces are a good idea, and we are planning on eventually allowing you to move older chunks to different tablespaces. Please feel free to reach out to us for further info on that functionality, since it's still being developed.

PS, make sure to use drop_chunks when you delete data so as not to create too many tombstones that end up having to be vacuumed.

  • I like the automatic rollups of old rrd-style databases. You resample historic time with average/min/max/quantiles with coarse resolution which greatly reduce storage need but still provides enough data (transparently) to see long term trends and patterns. – eckes Jul 20 at 4:42
  • Hey Diana. If you plan to be a regular you may like to have a think about this dba.meta.stackexchange.com/questions/2795/… – Michael Green Jul 20 at 10:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.