Skip to main content
replaced http://stackoverflow.com/ with https://stackoverflow.com/
Source Link

Since we are selecting most of the table, PostgreSQL chooses the sequential readingPostgreSQL chooses the sequential reading over the index and is right about it. This is the reason I used no index overall.

Since we are selecting most of the table, PostgreSQL chooses the sequential reading over the index and is right about it. This is the reason I used no index overall.

Since we are selecting most of the table, PostgreSQL chooses the sequential reading over the index and is right about it. This is the reason I used no index overall.

Source Link
Matjaž
  • 363
  • 2
  • 10

I run a test myself and figured out that changing the DB strcture helps a lot the performance of the first query and just a bit of the second one.

My configuration

My PostgreSQL configuration is default except the working memory, which is set to yours (set work_mem = '64MB'). I'm working on an SSD on a laptop with PostgreSQL 9.5.1. And I'm no expert!

New schematization

With this setup we put the tags in a separate table to easilly check they are unique.

CREATE TABLE IF NOT EXISTS tags (
    id  serial NOT NULL
  , tag text      NOT NULL UNIQUE
    
  , PRIMARY KEY (id)
    );
    
CREATE TABLE IF NOT EXISTS datapoints (
    fk_tag integer                     NOT NULL
  , time   timestamp without time zone NOT NULL
  , value  double precision

  , FOREIGN KEY (fk_tag)
        REFERENCES tags (id)
        ON UPDATE CASCADE  -- customize as needed
        ON DELETE RESTRICT -- customize as needed
    );

Populate tables

I filled the database with random data but 10x less than in your case or it would take me > 1 h to complete the inserts. Any result I will obtain from benchmarking, will also be multiplied by 10 to estimate a table of your size.

Notice! I do not know how many tags you have. It's just hypotetical.

INSERT INTO tags (tag)
    SELECT md5(random()::text) || md5(random()::text)
        FROM generate_series(1, 100000); -- Hypothetical number of tags!

INSERT INTO datapoints (fk_tag, time, value)
    SELECT trunc(random() * (SELECT max(id) - min(id) FROM tags) + (SELECT min(id) FROM tags) -- 99999 + 628940
        ,  current_timestamp
        ,  random()
        FROM generate_series(1,16272771);
        -- 10x less rows than you have.
        -- This INSERT took ~ 7 minutes to finish.

A little bit of cleanup before we start and we let the query planner gather some statistics about the tables.

VACUUM;
ANALYZE tags;
ANALYZE datapoints;

Comparing with your hardware

First of all I'll create a table like yours (still 10x smaller) to estimate the difference between my and your hardware speed.

SELECT t.tag
    ,  d.time
    ,  d.value
    INTO datapoints_joined
    FROM datapoints AS d
    INNER JOIN tags AS t
    ON d.fk_tag = t.id;

By running the two queries as you wrote them on datapoints_joined, the results 246.15766 s and 232.87162 s (already multiplied by 10) show that my computer is slower by 2.67 and 3.30 times, so approximately 3x slower.

Any result I will obtain from benchmarking, will also be divided by 3 to estimate a hardware of your speed. So basically it should be:

my execution time * 10/3 = your execution time

[I do really hope that all my estimates are ok...]

First query

SELECT tag FROM tags;

Since the tag column is already UNIQUE, there is no need for a DISTINCT.

Result

EXPLAIN ANALYZE SELECT tag FROM tags;

Seq Scan on tags  (cost=0.00..2235.00 rows=100000 width=65) (actual time=0.011..34.362 rows=100000 loops=1)
Planning time: 0.059 ms
Execution time: 42.998 ms -- SUCH SPEED, MUCH WOW!

On your machine (multipying the execution time by 10/3) it should take 143.33 ms = 0.14333 s which is ~519x faster than 74371.153 ms. Awesome!

Why is faster

In your case you were performing a DISTINCT query on the tag column which is varchar/text. This requires a checkup of the whole string each time.

In my case the column is declared as UNIQUE and is places in a separate table. This allows a simple select of all the tags which is obviously faster. The insertion of new tags, on the other hand, requires more because of the UNIQUE check.

Second query

Now we perform a JOIN to get the actual tag text instead of the foreign key, but we do it after the GROUP BY on the integer field fk_tag to increase the speed of the joining.

SELECT t.tag
    ,  d.minTime
    ,  d.maxTime
    FROM (
        SELECT fk_tag
            , min(time) AS minTime
            , max(time) AS maxTime
            FROM datapoints
            GROUP BY fk_tag
        ) AS d
    INNER JOIN tags AS t
    ON d.fk_tag = t.id;
            
Hash Join  (cost=391905.20..395209.83 rows=97915 width=81) (actual time=17849.685..17995.566 rows=99999 loops=1)
 Hash Cond: (datapoints.fk_tag = t.id)
 ->  HashAggregate  (cost=388420.20..389399.35 rows=97915 width=12) (actual time=17741.489..17807.022 rows=99999 loops=1)
       Group Key: datapoints.fk_tag
       ->  Seq Scan on datapoints  (cost=0.00..266375.40 rows=16272640 width=12) (actual time=0.006..2720.880 rows=16272771 loops=1)
 ->  Hash  (cost=2235.00..2235.00 rows=100000 width=69) (actual time=108.116..108.116 rows=100000 loops=1)
       Buckets: 131072  Batches: 1  Memory Usage: 11181kB
       ->  Seq Scan on tags t  (cost=0.00..2235.00 rows=100000 width=69) (actual time=0.011..46.569 rows=100000 loops=1)
Planning time: 0.241 ms
Execution time: 18005.401 ms -- MEH, NOT SO FAST

On your machine (multipying the execution time by 10/3) it should take 60018.00 ms = 60.018 s which is ~1.45x faster than 86922.564 ms. Not a huge result, but still better than nothing.

A few considerations

Since we are selecting most of the table, PostgreSQL chooses the sequential reading over the index and is right about it. This is the reason I used no index overall.

If you need the first query more than the second one, maybe this can be a solution for you. Please try it on your machine to be sure.