0

I have a table created like this:

-- enable timescale plugin
CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

-- create schema
CREATE SCHEMA IF NOT EXISTS {schemaName};

-- create table
CREATE TABLE IF NOT EXISTS {schemaName}.{tableName}
(
    ts           TIMESTAMP    NOT NULL,
    ticker       VARCHAR(16)  NOT NULL,
    m1           FLOAT4       NOT NULL,
    m5           FLOAT4       NOT NULL,
    m15          FLOAT4       NOT NULL,
    m30          FLOAT4       NOT NULL,
    h1           FLOAT4       NOT NULL,
    h2           FLOAT4       NOT NULL,
    h4           FLOAT4       NOT NULL,
    d1           FLOAT4       NOT NULL,
    high         FLOAT4       NOT NULL,
    vwap         FLOAT4       NOT NULL,
    low          FLOAT4       NOT NULL
);

-- create hypertable
SELECT create_hypertable('{schemaName}.{tableName}', 'ts', create_default_indexes => false, chunk_time_interval => INTERVAL '1 day', if_not_exists => TRUE);

-- set autovacum
ALTER TABLE {schemaName}.{tableName} SET (autovacuum_enabled = on);

-- create index
CREATE INDEX IF NOT EXISTS idx_kvwap_ticker_ts ON {schemaName}.{tableName}(ticker, ts DESC);

The data has one row per minute, but sometimes we need a large time span, for example one year. So I created views that do an aggregation of the data for different timeframes so we pull fewer data from the server.

The table has roughly 1.5M rows per ticker and about 140 tickers, for a total of about ~210M rows.

I create aggregate views like this:

let viewsCreationSQL =
    let intervals = [| oneDay; fourHours; twoHours; oneHour; thirtyMinutes; fifteenMinutes; fiveMinutes; oneMinute |]

    stringBuffer {
        for i in intervals do
            yield
                $"
                \n
                CREATE MATERIALIZED VIEW IF NOT EXISTS {schemaName}.kvwap_{i.ShortString.ToLower()}
                WITH (timescaledb.continuous) AS
                (
                    SELECT
                        time_bucket(INTERVAL '{i.TotalSeconds} second', ts) AS ts_bucket,
                        min(ticker)                  AS ticker,
                        round(avg(m1)::decimal,  8)  AS m1,
                        round(avg(m5)::decimal,  8)  AS m5,
                        round(avg(m15)::decimal, 8)  AS m15,
                        round(avg(m30)::decimal, 8)  AS m30,
                        round(avg(h1)::decimal,  8)  AS h1,
                        round(avg(h2)::decimal,  8)  AS h2,
                        round(avg(h4)::decimal,  8)  AS h4,
                        round(avg(d1)::decimal,  8)  AS d1,
                        min(low)                     AS low,
                        round(avg(vwap)::decimal, 8) AS vwap,
                        max(high)                    AS high
                    FROM
                        {schemaName}.kvwap
                    GROUP BY ticker, ts_bucket
                    ORDER BY ts_bucket
                );

                CREATE INDEX IF NOT EXISTS idx_kvwap_ticker_ts_{i.ShortString.ToLower()}
                ON {schemaName}.kvwap_{i.ShortString.ToLower()} (ticker, ts_bucket);
                "

    }

The views work as expected, but they are very slow. It takes about 2m to get the top 500 rows and will use a full CPU core during that time.

I thought the Timescale materialized views were creating tables behind the scenes, but it doesn't seem to be the case considering how slow the access is. Right now, pulling all the rows needed and aggregating them in the client performs similarly.

What am I missing?


edit:

I'm using latest-pg14 (docker image)

The command used is from the DataGrip viewer:

SELECT t.* FROM exchange_debug.kvwap_d1 t LIMIT 501

It really looks like the materialized view is being rebuilt for every query.

I can see that Timescale added triggers to the table:

create table if not exists exchange_debug.kvwap
(
    ts     timestamp   not null,
    ticker varchar(16) not null,
    m1     real        not null,
    m5     real        not null,
    m15    real        not null,
    m30    real        not null,
    h1     real        not null,
    h2     real        not null,
    h4     real        not null,
    d1     real        not null,
    high   real        not null,
    vwap   real        not null,
    low    real        not null
)
    with (autovacuum_enabled = on);

alter table exchange_debug.kvwap
    owner to root;

create index if not exists idx_kvwap_ticker_ts
    on exchange_debug.kvwap (ticker asc, ts desc);

create trigger ts_insert_blocker
    before insert
    on exchange_debug.kvwap
    for each row
execute procedure ???();

create trigger ts_cagg_invalidation_trigger
    after insert or update or delete
    on exchange_debug.kvwap
    for each row
execute procedure ???('115');
2
  • What version of Timescale are you running? And what is the query you're running to get the "top 500 rows?
    – David K
    Feb 5, 2023 at 20:47
  • @davidk, I've added more info to the question
    – Thomas
    Feb 14, 2023 at 9:06

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.