1

I just setup a new AWS RDS postgres server and queries are painfully slow (> 10 hours) such that most of the time I just end up killing the query. I have done some indexing that cut the EXPLAIN cost down by a factor of ten but it is still just as slow. This is a migration from redshift where the same query used to take about 5minutes to complete so I would've expected this to be around the same time since redshift only has one sort key.

Running AWS RDS Postgres 9.5.2 on a db.r3.8xlarge (32vCPU, 244gb RAM), work_mem=1GB.

Query

SELECT *
FROM 
    (SELECT *
    FROM table1 
    WHERE create_date >= '2015-10-01' AND create_date <= '2015-12-31')  
JOIN 
    (SELECT  * 
    FROM table2 
    WHERE type = 6)  
    ON table1.id = table2.id 
WHERE table1.indicator1 = 'N' AND table1.product = ‘foo’ AND table1.indicator2 = ‘Y’

EXPLAIN Output

 Nested Loop  (cost=109063.49..593806.97 rows=20886 width=91)
   ->  Bitmap Heap Scan on table1  (cost=109062.91..255339.43 rows=39386 width=80)
         Recheck Cond: (((product)::text = ‘foo’::text) AND (indicator2 = ‘Y’::bpchar) AND (indicator1 = 'N'::bpchar) AND (create_date >= '2015-10-01'::date) AND (create_date <= '2015-12-31'::date))
         ->  BitmapAnd  (cost=109062.91..109062.91 rows=39386 width=0)
               ->  Bitmap Index Scan on multi_no_date_idx  (cost=0.00..22893.13 rows=662445 width=0)
                     Index Cond: (((product)::text = ‘foo’::text) AND (indicator2 = ‘Y’::bpchar) AND (indicator1 = 'N'::bpchar))
               ->  Bitmap Index Scan on create_date_idx  (cost=0.00..86149.84 rows=4108527 width=0)
                     Index Cond: ((create_date >= '2015-10-01'::date) AND (create_date <= '2015-12-31'::date))
   ->  Index Scan using table2_pkey on table2  (cost=0.57..8.57 rows=1 width=19)
         Index Cond: ((id = table1.id) AND (type = 1))
(10 rows)

Table1 Indices

Indexes:
    “table1_pkey" PRIMARY KEY, btree (id)
    “indicator1_idx" btree (indicator1)
    “create_date_idx" btree (create_date)
    “indicator2” btree (indicator2)
    “table1_id_idx" btree (id)
    “product_idx” btree (product)
    "multi_no_date_idx" btree (indicator1, indicator2, product)

Table2 Indices

Indexes:
    “table2_pkey" PRIMARY KEY, btree (id, type)
    “id_idx" btree (id)
3
  • The subqueries make no sense. I think the PostgreSQL optimizer is smart enough to figure this out, but it can be worth a try to remove them, joining the two tables, not the subqueries. Also, could you show the query plan generated by RedShift? Commented Oct 24, 2016 at 19:59
  • Queries are autogenerated by python blaze. I agree maybe not the most ideal but postgres seems to optimize for it appropriately according to the query plan. Even still I wouldn't think it would take longer than a day let alone 30min.
    – postelrich
    Commented Oct 24, 2016 at 20:19
  • Redshift: XN Hash Join DS_DIST_NONE (cost=7840.77..20461918.52 rows=1 width=312) Hash Cond: ("outer".id = "inner".id) -> XN Seq Scan on table2 (cost=0.00..20276990.40 rows=14166984 width=27) Filter: (type = 1) -> XN Hash (cost=7840.77..7840.77 rows=1 width=298) -> XN Seq Scan on table1 (cost=0.00..7840.77 rows=1 width=298) Filter: ((create_date <= '2015-12-31'::date) AND (create_date >= '2015-10-01'::date) AND ((product)::text = ‘foo’::text) AND (indicator2 = ‘Y’::bpchar) AND (indicator1 = 'N'::bpchar))
    – postelrich
    Commented Oct 24, 2016 at 20:20

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.