0

The following query execution too long in statistics state and I couldn't figure out why.

DB engine - 5.7.mysql_aurora.2.07.2

DB Size - db.r5.4xlarge

Sample Query Profile output

+--------------------------------+----------+
| Status                         | Duration |
+--------------------------------+----------+
| starting                       | 0.000023 |
| checking query cache for query | 0.000155 |
| checking permissions           | 0.000009 |
| checking permissions           | 0.000002 |
| checking permissions           | 0.000003 |
| checking permissions           | 0.000002 |
| checking permissions           | 0.000009 |
| Opening tables                 | 0.000035 |
| init                           | 0.000102 |
| System lock                    | 0.000035 |
| optimizing                     | 0.000004 |
| optimizing                     | 0.000003 |
| optimizing                     | 0.000011 |
| statistics                     | 0.224528 |
| preparing                      | 0.000030 |
| Sorting result                 | 0.000017 |
| statistics                     | 0.000041 |
| preparing                      | 0.000013 |
| Creating tmp table             | 0.000023 |
| optimizing                     | 0.000013 |
| statistics                     | 0.064207 |
| preparing                      | 0.000035 |
| Sorting result                 | 0.000025 |
| statistics                     | 0.000098 |
| preparing                      | 0.000018 |
| executing                      | 0.000011 |
| Sending data                   | 0.000007 |
| executing                      | 0.000003 |
| Sending data                   | 0.000251 |
| executing                      | 0.000007 |
| Sending data                   | 0.000003 |
| executing                      | 0.000002 |
| Sending data                   | 0.000526 |
| end                            | 0.000007 |
| query end                      | 0.000013 |
| removing tmp table             | 0.000007 |
| query end                      | 0.000004 |
| closing tables                 | 0.000003 |
| removing tmp table             | 0.000004 |
| closing tables                 | 0.000002 |
| removing tmp table             | 0.000005 |
| closing tables                 | 0.000002 |
| removing tmp table             | 0.000004 |
| closing tables                 | 0.000010 |
| freeing items                  | 0.000050 |
| storing result in query cache  | 0.000007 |
| cleaned up                     | 0.000004 |
| cleaning up                    | 0.000017 |
+--------------------------------+----------+

Query

select xo.ITEM, xo.VALUE
from (
         select pi.ITEM, pi.ITEM_GROUP, pi.VALUE
         from TABLE_2 pi
                  inner join (select max(ps.EXPORTED_DATE) as max_expo, ps.ITEM
                              from TABLE_2 ps
                                       inner join (
                                  select max(pp.EFFECTIVE_DATE) max_eff_TABLE_2, pp.ITEM
                                  from TABLE_2 pp
                                  where pp.EFFECTIVE_DATE <= '2020/07/17'
                                    and ITEM in
                                        ('20', '30', '40', '50', '110', '120', '320', '520', '720', '820', '920', '321',
                                         '275', '221')
                                  group by ITEM
                              ) a on ps.EFFECTIVE_DATE = a.max_eff_TABLE_2 and ps.ITEM = a.ITEM
                              group by a.ITEM) rr on rr.ITEM = pi.ITEM and rr.max_expo = pi.EXPORTED_DATE) xo

         inner join (
    select ea.ITEM, ea.CUSTOMER_ID, ea.ITEM_GROUP
    from TABLE_1 ea
             inner join (
        select MAX(e.EFFECTIVE_DATE) eat_max_eff, e.ITEM, e.CUSTOMER_ID
        from TABLE_1 e
        where e.CUSTOMER_ID = '20'
          and ITEM in ('20', '30', '40', '50', '110', '120', '320', '520', '720', '820', '920', '321', '275', '221')
          and EFFECTIVE_DATE <= '2020/07/17'
        group by e.ITEM
    ) aa
    where ea.ITEM = (aa.ITEM)
      and ea.CUSTOMER_ID = aa.CUSTOMER_ID
      and ea.EFFECTIVE_DATE = aa.eat_max_eff) lo
                    on lo.ITEM_GROUP = xo.ITEM_GROUP and lo.ITEM = xo.ITEM;

Indexes

Table 1

mysql> SHOW INDEX FROM T1;
+-------+------------+--------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name     | Seq_in_index | Column_name    | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment |
+-------+------------+--------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| T1    |          0 | PRIMARY      |            1 | CUSTOMER_ID    | A         |     3297549 |     NULL | NULL   |      | BTREE      |         |               |
| T1    |          0 | PRIMARY      |            2 | ITEM           | A         |   687374784 |     NULL | NULL   |      | BTREE      |         |               |
| T1    |          0 | PRIMARY      |            3 | EFFECTIVE_DATE | A         |  1314196480 |     NULL | NULL   |      | BTREE      |         |               |
| T1    |          1 | t1_ix_item   |            1 | ITEM           | A         |     2151649 |     NULL | NULL   |      | BTREE      |         |               |
+-------+------------+--------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

Table 2


mysql> SHOW INDEX FROM TABLE_2;
+-------+------------+-----------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| Table | Non_unique | Key_name              | Seq_in_index | Column_name    | Collation | Cardinality | Sub_T2rt | T2cked | Null | Index_type | Comment | Index_comment |
+-------+------------+-----------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+
| T2    |          0 | PRIMARY               |            1 | ITEM           | A         |           1 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          0 | PRIMARY               |            2 | ITEM_GROUP     | A         |       14265 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          0 | PRIMARY               |            3 | EFFECTIVE_DATE | A         |    63663076 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          0 | PRIMARY               |            4 | EXPORTED_DATE  | A         |    62464764 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_expo       |            1 | ITEM           | A         |      115823 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_expo       |            2 | EXPORTED_DATE  | A         |    13766454 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_eff_date   |            1 | ITEM           | A         |      115823 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_eff_date   |            2 | EFFECTIVE_DATE | A         |    13766454 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_eff_ig     |            1 | ITEM           | A         |      115823 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_eff_ig     |            2 | EFFECTIVE_DATE | A         |    13766454 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_ix_item_eff_ig     |            3 | ITEM_GROUP     | A         |    68216912 |     NULL | NULL   |      | BTREE      |         |               |
| T2    |          1 | t2_idx_effective_date |            1 | EFFECTIVE_DATE | A         |       79406 |     NULL | NULL   |      | BTREE      |         |               |
+-------+------------+-----------------------+--------------+----------------+-----------+-------------+----------+--------+------+------------+---------+---------------+

According to this: statistics State in MySQL Processlist

I checked the innodb_buffer_pool_size.

mysql> SHOW VARIABLES LIKE "innodb_buffer_pool_size";
+-------------------------+-------------+
| Variable_name           | Value       |
+-------------------------+-------------+
| innodb_buffer_pool_size | 96223625216 |
+-------------------------+-------------+

In EXPLAIN output rows are minimal (Depends on the Item count in the query. If Item count is 10, the number of rows were 20). Even though the row counts are minimal why the query takes too long in statistics state?

Improve this question

1 Answer 1

Reset to default
0

This is "wrong":

            SELECT  MAX(e.EFFECTIVE_DATE) eat_max_eff, e.ITEM, e.CUSTOMER_ID
                from  TABLE_1 e
                where  e.CUSTOMER_ID = '20'
                  and  ITEM in (...)
                  and  EFFECTIVE_DATE <= '2020/07/17'
                group by  e.ITEM

See "only_full_group_by" for discussion. In particular, e.CUSTOMER_ID will _not necessarily be the id for the MAX(e.EFFECTIVE_DATE) that was found. Please fix that "groupwise-max" problem before further discussion of "statistics".

"Profile" is not the best of tools. Suggest getting more info via EXPLAIN FORMAT=JSON and "Optimizer trace".

Improve this answer
3
  • Since I'm choosing the EFFECTIVE_DATE of the rows which satisfies the where clause of e.CUSTOMER_ID = '20', always the SELECT e.CUSTOMER_ID will be 20, isn't it?
    – Vithulan
    Commented Jun 29, 2020 at 18:05
  • @Vithulan - Yes, customer_id will always be 20. However, item is not a single value. One index can help with 2 things in the where; another index can help with a diff 2. But there is no possible index to handle all 3 columns efficiently. (Or, am I missing your point?)
    – Rick James
    Commented Jun 29, 2020 at 18:39
  • The primary key of that table is customer_id, item and 'effective_date`. That's what I use now but not sure how efficient that is.
    – Vithulan
    Commented Jun 30, 2020 at 2:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.