61

I'm trying to optimize a query which looks into a big table (37 millions rows) and have a question about what order the operations are executed in a query.

select 1 
from workdays day
where day.date_day >= '2014-10-01' 
    and day.date_day <= '2015-09-30' 
    and day.offer_id in (
        select offer.offer_day 
        from offer  
        inner join province on offer.id_province = province.id_province  
        inner join center cr on cr.id_cr = province.id_cr 
        where upper(offer.code_status) <> 'A' 
            and province.id_region in ('10' ,'15' ,'21' ,'26' ,'31' , ...,'557') 
            and province.id_cr in ('9' ,'14' ,'20' ,'25' ,'30' ,'35' ,'37')
    )

Are the WHERE clauses for the date range executed before the subquery? Is it a good way to put the most restrictive clauses first to avoid big loops for other clauses, in order to do a faster execution?

Now queries takes so much time to execute.

0

2 Answers 2

102

To elaborate on @alci's answer:

PostgreSQL doesn't care what order you write things in

  • PostgreSQL doesn't care at all about the order of entries in a WHERE clause, and chooses indexes and execution order based on cost and selectivity estimation alone.

  • The order in which joins are written is also ignored up to the configured join_collapse_limit; if there are more joins than that, it'll execute them in the order they're written.

  • Subqueries can be executed before or after the query that contains them, depending on what's fastest, so long as the subquery is executed before the outer query actually needs the information. Often in reality the subquery gets executed kind-of in the middle, or interleaved with the outer query.

  • There's no guarantee PostgreSQL will actually execute parts of the query at all. They can be completely optimized away. This is important if you call functions with side-effects.

PostgreSQL will transform your query

PostgreSQL will heavily transform queries while retaining the exact same effects, in order to make them run faster while not changing the results.

  • Terms outside a subquery can get pushed down into the subquery so they execute as part of the subquery not where you wrote them in the outer query

  • Terms in the subquery can be pulled up to the outer query so their execution is done as part of the outer query, not where you wrote them in the subquery

  • The subquery can, and often is, flattened into a join on the outer table. The same is true of things like EXISTS and NOT EXISTS queries.

  • Views get flattened into the query that uses the view

  • SQL functions often get inlined into the calling query

  • ... and there are numerous other transformations made to queries, such as constant expression pre-evaluation, de-correlation of some subqueries, and all sorts of other planner/optimizer tricks.

In general PostgreSQL can massively transform and rewrite your query, to the point where each of these queries:

select my_table.*
from my_table
left join other_table on (my_table.id = other_table.my_table_id)
where other_table.id is null;

select *
from my_table
where not exists (
  select 1
  from other_table
  where other_table.my_table_id = my_table.id
);

select *
from my_table
where my_table.id not in (
  select my_table_id
  from other_table
  where my_table_id is not null
);

will usually all produce exactly the same query plan. (Assuming I didn't make any dumb mistakes in the above anyway).

It's not uncommon to try to optimize a query only to find that the query planner has already figured out the tricks you're trying and applied them automatically, so the hand-optimized version is no better than the original.

Limitations

The planner/optimizer is far from omnicient, and is limited by the requirement to be absolutely certain it can't change the effects of the query, the available data to make decisions with, the rules that've been implemented, and the CPU time it can afford to spend pondering the optimizations. For example:

  • The planner relies on statistics kept by ANALYZE (usually via autovacuum). If these are outdated, the plan choice can be bad.

  • The statistics are only a sample, so they can be misleading due to sampling effects, especially if too small a sample is taken. Bad plan choices can result.

  • The statistics don't keep track of some kinds of data about the table, like correlations between columns. This can lead the planner to make bad decisions when it assumes things are independent when they aren't.

  • The planner relies on cost parameters like random_page_cost to tell it the relative speed of various operations on the particular system it's installed on. These are only guides. If they're badly wrong they can lead to poor plan choices.

  • Any subquery with a LIMIT or OFFSET cannot be flattened or be subject to pullup / pushdown. This doesn't mean it'll execute before all parts of the outer query, though, or even that it'll execute at all.

  • CTE terms (the clauses in a WITH query) are always executed in their entirety, if they're executed at all. They can't be flattened, and terms can't be pushed up or pulled down across the CTE term barrier. CTE terms are always executed before the final query. This is non-SQL-standard behaviour, but it's documented as how PostgreSQL does things.

  • PostgreSQL has a limited ability to optimize across queries on foreign tables, security_barrier views, and certain other special kinds of relation

  • PostgreSQL won't inline a function written in anything except plain SQL, nor do pullup/pushdown between it and the outer query.

  • The planner/optimizer is really dumb about selecting expression indexes, and about trivial data type differences between index and expression.

Tons more, too.

Your query

In the case of your query:

select 1 
from workdays day
where day.date_day >= '2014-10-01' 
    and day.date_day <= '2015-09-30' 
    and day.offer_id in (
        select offer.offer_day 
        from offer  
        inner join province on offer.id_province = province.id_province  
        inner join center cr on cr.id_cr = province.id_cr 
        where upper(offer.code_status) <> 'A' 
            and province.id_region in ('10' ,'15' ,'21' ,'26' ,'31' , ...,'557') 
            and province.id_cr in ('9' ,'14' ,'20' ,'25' ,'30' ,'35' ,'37')
    )

nothing stops it from being flattened into a simpler query with an extra set of joins, and it very likely will be.

It'll probably turn out something like (untested, obviously):

select 1 
from workdays day
inner join offer on day.offer_id = offer.offer_day
inner join province on offer.id_province = province.id_province  
inner join center cr on cr.id_cr = province.id_cr 
where upper(offer.code_status) <> 'A' 
   and province.id_region in ('10' ,'15' ,'21' ,'26' ,'31' , ...,'557') 
   and province.id_cr in ('9' ,'14' ,'20' ,'25' ,'30' ,'35' ,'37')
   and day.date_day >= '2014-10-01' 
   and day.date_day <= '2015-09-30';

PostgreSQL will then optimise the join order and join methods based on its selectivity and row count estimates and the available indexes. If these reasonably reflect reality then it'll do the joins and run the where clause entries in whatever order is best - often mixing them together, so it does a bit of this, then a bit of that, then comes back to the first part, etc.

How to see what the optimizer did

You can't see the SQL that PostgreSQL optimizes your query into, because it converts the SQL to an internal query tree representation then modifies that. You can dump the query plan and compare it to other queries.

There's no way to "deparse" that query plan or the internal plan tree back to SQL.

http://explain.depesz.com/ has a decent query plan helper. If you're totally new to query plans etc (in which case I'm amazed you made it this far through this post) then PgAdmin has a graphical query plan viewer that provides much less information but is simpler.

Related reading:

Pushdown/pullup and flattening capabilities continue to improve in each release. PostgreSQL is usually right about pull-up/push-down/flattening decisions, but not always, so occasionally you have to (ab)use a CTE or the OFFSET 0 hack. If you find such a case, report a query planner bug.


If you're really, really keen you can also use the debug_print_plans option to see the raw query plan, but I promise you don't want to read that. Really.

5
  • What about the associativity / nesting of where subclauses? For example, does WHERE A AND (B AND C) result in exactly the same execution plan as WHERE (A AND B) AND C? Commented Jan 9 at 0:22
  • 1
    @LukeHutchison postgres resolves operator precedence early, before the optimiser pass. It'll completely ignore your forced ordering if the operations are commutative since it knows that ordering doesn't matter. So you can't use that to do things like short-circuit execution of an expression that might raise an error; writing WHERE x = 0 OR (10%1 = 1) will NOT prevent zero-divide errors for example. Constructs like CASE can handle situations where strictly forced ordering is required. Commented Mar 9 at 7:04
  • thanks, that's surprising that short-circuit evaluation doesn't work in queries, but given that, at least I would assume that A AND B AND C should be treated the same as C AND B AND A. It's not as clear to me that this should extend beyond commutativity to associativity. Does Postgres actually flatten out the operator tree when operators have the same precedence level? i.e. are (A AND B) AND C and A AND (B AND C) treated the same as A AND B AND C? Commented Mar 10 at 8:49
  • 1
    @LukeHutchison Yes. Since AND is commutative and there's no short-circuit boolean evaluation in SQL, those are all logically the same thing. Associativity isn't really relevant in a chain of AND operators. For mixed AND and OR, it'll respect your forced associativity since the semantics are different - A OR (B AND C) is semantically different to (A OR B) AND C so Pg will preserve that - but it may (or may not, depending on optimiser choices) actually run C if one of A or B is false, or vice versa. Commented Mar 13 at 0:47
  • 1
    So the point is that whether you write A AND (B AND C) or (A AND B) AND C or A AND B AND C, postgres may run any or all of the expression terms, in any order, and you cannot make any assumptions about them for the purposes of side-effects or exceptional-value handling. It might run C and if it's false, skip running A and B. It might run C, B and A independently, in that order or any other, even if one is false. You can't assume anything unless you use constructs like CASE. Commented Mar 13 at 0:53
23

SQL is a declarative language: you tell what you want, not how to do it. The RDBMS will choose the way it will execute the query, called the execution plan.

Once upon a time (5-10 years ago), the way a query was written had a direct impact on the execution plan, but nowadays, most SQL database engines use a Cost Based Optimizer for planning. That is, it will evaluate different strategies to execute the query, based on its statistics on the database objects, and choose the best one.

Most of the time, it's really the best one, but sometimes the DB engine will make bad choices, resulting in very slow queries.

3
  • It should be noted that on some RDBMSs query order is still significant, but for the more advanced ones everything you say is true in practise as well as theory. When the query planner is picking bad choices of execution order there are usually query hints available to push it in a more efficient direction (such as WITH(INDEX(<index>)) in MSSQL to force the choice of index for a particular join). Commented Sep 22, 2015 at 8:03
  • The question is if some index on date_day actually exists. If there is none then the optimizer does not have many plans to compare.
    – jkavalik
    Commented Sep 22, 2015 at 10:01
  • There can still be quirks though. For example, the order in which you put tables in a query in postgres affects order of join execution if the number of joins is greater than the configured join_collapse_limit. And in older postgres versions, using WITH (...) instead of FROM SELECT (...) would produce different query plans, as WITH used to implicitly impose an optimization barrier. Commented Mar 9 at 7:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.