0

When I run the following SELECT, it takes quite long, typically 30 or 40 minutes (I'm on Debian, MySQL 5.5.62):

SELECT ga.currency, o.organisation_name, os.hostname, g.global_name,
       SUM(IF(ga.type ='WAGER', ga.amount,0)) AS 'total wager lc',
       SUM(IF(ga.type ='WAGER', ga.amount,0)/fx.gbp) AS 'total wager gbp',
       SUM(IF(ga.type ='WAGER', ga.amount,0)/fx.eur) AS 'total wager eur',
       SUM(IF(ga.type ='WAGER', ga.amount,0)/fx.usd) AS 'total wager usd',
       SUM(IF(ga.type = 'WIN', ga.amount, 0)) AS 'total win lc',
       SUM(IF(ga.type = 'WIN', ga.amount, 0)/fx.gbp) AS 'total win gbp',
       SUM(IF(ga.type = 'WIN', ga.amount, 0)/fx.eur) AS 'total win eur',
       SUM(IF(ga.type = 'WIN', ga.amount, 0)/fx.usd) AS 'total win usd',
       CONCAT( DATE(ga.created_timestamp),' ', HOUR(ga.created_timestamp)) AS date_and_hour
 FROM game_action ga
 JOIN user u               ON u.user_id = ga.user_id
 JOIN game g               ON g.game_id = ga.game_id
 JOIN game_instance gi     ON gi.game_instance_id = ga.game_instance_id
 JOIN organisation_site os ON u.organisation_site_id = os.organisation_site_id
 JOIN organisation o       ON o.organisation_id = os.organisation_id
 INNER JOIN reporting.fx_conversion fx ON fx.date = date(ga.created_timestamp) 
                                      AND fx.currency = ga.currency

WHERE ga.created_timestamp BETWEEN str_to_date(CONCAT(DATE_FORMAT(DATE_SUB(NOW(), INTERVAL 1 WEEK),'%Y-%m-%d'),' 00:00:00'),'%Y-%m-%d %T')
  AND str_to_date(CONCAT(DATE_SUB(LAST_DAY(DATE_SUB(NOW(), INTERVAL 1 WEEK)), INTERVAL -1 DAY),' 23:59:59'),'%Y-%m-%d %T')
  AND gi.status IN ('RESOLVED', 'AUTO_COMPLETED')

GROUP BY ga.currency
       , o.organisation_name
       , os.hostname
       , g.global_name
       , date_and_hour

ORDER BY o.organisation_name
       , os.hostname
       , ga.created_timestamp ASC
;

However, the explain suggests that this shouldn't happen (or I don't understand how things work):

mysql> explain extended ...
+----+-------------+-------+--------+-----------------------------------------------------------------------+---------+---------+--------------------------------+----------+----------+----------------------------------------------+
| id | select_type | table | type   | possible_keys                                                         | key     | key_len | ref                            | rows     | filtered | Extra                                        |
+----+-------------+-------+--------+-----------------------------------------------------------------------+---------+---------+--------------------------------+----------+----------+----------------------------------------------+
|  1 | SIMPLE      | ga    | ALL    | GA_IX01,GA_IX02,GA_IX03,game_action_created_timestamp,ga_id_cur_tstmp | NULL    | NULL    | NULL                           | 23851088 |    49.75 | Using where; Using temporary; Using filesort |
|  1 | SIMPLE      | g     | eq_ref | PRIMARY                                                               | PRIMARY | 4       | gameiom.ga.game_id             |        1 |   100.00 |                                              |
|  1 | SIMPLE      | fx    | ref    | PRIMARY                                                               | PRIMARY | 3       | func                           |       22 |   100.00 | Using where                                  |
|  1 | SIMPLE      | u     | eq_ref | PRIMARY,U_UK01,U_IX_04                                                | PRIMARY | 4       | gameiom.ga.user_id             |        1 |   100.00 |                                              |
|  1 | SIMPLE      | os    | eq_ref | PRIMARY,ORGST_UK1_orgid_extid,ORGST_FK01                              | PRIMARY | 4       | gameiom.u.organisation_site_id |        1 |   100.00 |                                              |
|  1 | SIMPLE      | o     | eq_ref | PRIMARY                                                               | PRIMARY | 4       | gameiom.os.organisation_id     |        1 |   100.00 |                                              |
|  1 | SIMPLE      | gi    | ref    | PRIMARY,game_instance_status                                          | PRIMARY | 8       | gameiom.ga.game_instance_id    |        1 |   100.00 | Using where                                  |
+----+-------------+-------+--------+-----------------------------------------------------------------------+---------+---------+--------------------------------+----------+----------+----------------------------------------------+
7 rows in set, 1 warning (0.00 sec)

mysql> show warnings\G
*************************** 1. row ***************************
  Level: Note
   Code: 1003
Message: select `gameiom`.`ga`.`currency` AS `currency`,`gameiom`.`o`.`organisation_name` AS `organisation_name`,`gameiom`.`os`.`hostname` AS `hostname`,`g`.`global_name` AS `global_name`,sum(if((`gameiom`.`ga`.`type` = 'WAGER'),`gameiom`.`ga`.`amount`,0)) AS `total wager lc`,sum((if((`gameiom`.`ga`.`type` = 'WAGER'),`gameiom`.`ga`.`amount`,0) / `reporting`.`fx`.`gbp`)) AS `total wager gbp`,sum((if((`gameiom`.`ga`.`type` = 'WAGER'),`gameiom`.`ga`.`amount`,0) / `reporting`.`fx`.`eur`)) AS `total wager eur`,sum((if((`gameiom`.`ga`.`type` = 'WAGER'),`gameiom`.`ga`.`amount`,0) / `reporting`.`fx`.`usd`)) AS `total wager usd`,sum(if((`gameiom`.`ga`.`type` = 'WIN'),`gameiom`.`ga`.`amount`,0)) AS `total win lc`,sum((if((`gameiom`.`ga`.`type` = 'WIN'),`gameiom`.`ga`.`amount`,0) / `reporting`.`fx`.`gbp`)) AS `total win gbp`,sum((if((`gameiom`.`ga`.`type` = 'WIN'),`gameiom`.`ga`.`amount`,0) / `reporting`.`fx`.`eur`)) AS `total win eur`,sum((if((`gameiom`.`ga`.`type` = 'WIN'),`gameiom`.`ga`.`amount`,0) / `reporting`.`fx`.`usd`)) AS `total win usd`,concat(cast(`gameiom`.`ga`.`created_timestamp` as date),' ',hour(`gameiom`.`ga`.`created_timestamp`)) AS `date_and_hour` from `gameiom`.`game_action` `ga` join `gameiom`.`user` `u` join `gameiom`.`game` `g` join `gameiom`.`game_instance` `gi` join `gameiom`.`organisation_site` `os` join `gameiom`.`organisation` `o` join `reporting`.`fx_conversion` `fx` where ((`gameiom`.`u`.`user_id` = `gameiom`.`ga`.`user_id`) and (`g`.`game_id` = `gameiom`.`ga`.`game_id`) and (`gameiom`.`os`.`organisation_site_id` = `gameiom`.`u`.`organisation_site_id`) and (`gameiom`.`o`.`organisation_id` = `gameiom`.`os`.`organisation_id`) and (`gameiom`.`ga`.`created_timestamp` between <cache>(str_to_date(concat(date_format((now() - interval 1 week),'%Y-%m-%d'),' 00:00:00'),'%Y-%m-%d %T')) and <cache>(str_to_date(concat((last_day((now() - interval 1 week)) - interval -(1) day),' 23:59:59'),'%Y-%m-%d %T'))) and (`gameiom`.`gi`.`status` in ('RESOLVED','AUTO_COMPLETED')) and (`reporting`.`fx`.`date` = cast(`gameiom`.`ga`.`created_timestamp` as date)) and (convert(`reporting`.`fx`.`currency` using utf8) = `gameiom`.`ga`.`currency`) and (`gameiom`.`gi`.`game_instance_id` = `gameiom`.`ga`.`game_instance_id`)) group by `gameiom`.`ga`.`currency`,`gameiom`.`o`.`organisation_name`,`gameiom`.`os`.`hostname`,`g`.`global_name`,concat(cast(`gameiom`.`ga`.`created_timestamp` as date),' ',hour(`gameiom`.`ga`.`created_timestamp`)) order by `gameiom`.`o`.`organisation_name`,`gameiom`.`os`.`hostname`,`gameiom`.`ga`.`created_timestamp`
1 row in set (0.00 sec)

Running the SELECT with profiling shows:

mysql> show profile;
+--------------------------------+------------+
| Status                         | Duration   |
+--------------------------------+------------+
| starting                       |   0.000013 |
| Waiting for query cache lock   |   0.000003 |
| Waiting on query cache mutex   |   0.000002 |
| checking query cache for query |   0.000147 |
| checking permissions           |   0.000003 |
| checking permissions           |   0.000002 |
| checking permissions           |   0.000002 |
| checking permissions           |   0.000002 |
| checking permissions           |   0.000002 |
| checking permissions           |   0.000002 |
| checking permissions           |   0.000004 |
| Opening tables                 |   0.000046 |
| System lock                    |   0.000065 |
| init                           |   0.000078 |
| optimizing                     |   0.000046 |
| statistics                     |   0.000157 |
| preparing                      |   0.000026 |
| Creating tmp table             |   0.000199 |
| executing                      |   0.000003 |
| Copying to tmp table           | 999.999999 |
| Sorting result                 |   0.034862 |
| Sending data                   |   0.019785 |
| end                            |   0.000005 |
| removing tmp table             |   0.000379 |
| end                            |   0.000005 |
| query end                      |   0.000005 |
| closing tables                 |   0.000080 |
| freeing items                  |   0.000040 |
| logging slow query             |   0.000002 |
| logging slow query             |   0.000111 |
| cleaning up                    |   0.000006 |
+--------------------------------+------------+
31 rows in set (0.00 sec)

Now, I have to admit that I don't understand the warning after the EXPLAIN, and perhaps it refers to something that causes the tmp table (?). However, that aside, is the SQL shaped such that it will lead to tmp tables? What I've found in the docs is a bit hard to follow.

Edit:

Sorr, I had a couple of urgent things to do yesterday, but I've now had time to re-visit this subject. I tried to run explain on my select one more time, and now it looks a lot more like I thought I remembered it:

+----+-------------+-------+--------+------------------------------------------+-----------------------+---------+---------------------------------+------+----------+---------------------------------+
| id | select_type | table | type   | possible_keys                            | key                   | key_len | ref                             | rows | filtered | Extra                           |
+----+-------------+-------+--------+------------------------------------------+-----------------------+---------+---------------------------------+------+----------+---------------------------------+
|  1 | SIMPLE      | o     | ALL    | PRIMARY                                  | NULL                  | NULL    | NULL                            |    2 |   100.00 | Using temporary; Using filesort |
|  1 | SIMPLE      | os    | ref    | PRIMARY,ORGST_UK1_orgid_extid,ORGST_FK01 | ORGST_UK1_orgid_extid | 4       | gameiom.o.organisation_id       |    5 |   100.00 |                                 |
|  1 | SIMPLE      | u     | ref    | PRIMARY,U_UK01,U_IX_04                   | U_IX_04               | 4       | gameiom.os.organisation_site_id | 1566 |   100.00 | Using index                     |
|  1 | SIMPLE      | ga    | ref    | GA_IX01,GA_IX02,GA_IX03                  | GA_IX02               | 4       | gameiom.u.user_id               |  287 |   100.00 | Using where                     |
|  1 | SIMPLE      | g     | eq_ref | PRIMARY                                  | PRIMARY               | 4       | gameiom.ga.game_id              |    1 |   100.00 |                                 |
|  1 | SIMPLE      | gi    | ref    | PRIMARY                                  | PRIMARY               | 8       | gameiom.ga.game_instance_id     |    1 |   100.00 | Using where                     |
|  1 | SIMPLE      | fx    | ref    | PRIMARY                                  | PRIMARY               | 3       | func                            |    5 |   100.00 | Using where                     |
+----+-------------+-------+--------+------------------------------------------+-----------------------+---------+---------------------------------+------+----------+---------------------------------+
7 rows in set, 1 warning (0.00 sec)

And in fact, I copied the SELECT from what I had uploaded at the beginning of my question. The only explanation I can think of is that I must have copied the wrong output. Sorry about that. The profiling data I uploaded should be right, but I'll re-run just to make sure and update if there's any change.

  • I don't understand the warning after the EXPLAIN It is query text with fully qualified objects names, not more. Do not pay attention. – Akina Jan 15 at 12:21
  • You have an ORDER BY in your statement, at least in MSSQL that usually necessitates a worktable (i.e. temp db). And if your result set is large then it may spill to temp anyways. You only have 31 rows in your output... but how many are processed before being grouped? – Jonathan Fite Jan 15 at 13:13
  • @ypercubeᵀᴹ - oh, that escaped my attention. I thought I had got rid of the full table scan. Let me see if I can find that again. – j4nd3r53n Jan 15 at 14:27
  • For mysql both INNER JOIN .. ON and JOIN .. ON are equivalent. – Kondybas Jan 15 at 15:23
  • The 'warning' after EXPLAIN shows the query after parsing and name resolution. Sometimes it points out things like shifting ON clauses to the WHERE clause. – Rick James Jan 18 at 23:02
1

It is not surprising that temp tables would come into play. Why ???

According to the MySQL 5.5 Documentation on join_buffer_size (Italics mine):

The minimum size of the buffer that is used for plain index scans, range index scans, and joins that do not use indexes and thus perform full table scans. Normally, the best way to get fast joins is to add indexes. Increase the value of join_buffer_size to get a faster full join when adding indexes is not possible. One join buffer is allocated for each full join between two tables. For a complex join between several tables for which indexes are not used, multiple join buffers might be necessary.

There is no gain from setting the buffer larger than required to hold each matching row, and all joins allocate at least the minimum size, so use caution in setting this variable to a large value globally. It is better to keep the global setting small and change to a larger setting only in sessions that are doing large joins. Memory allocation time can cause substantial performance drops if the global size is larger than needed by most queries that use it.

The maximum permissible setting for join_buffer_size is 4GB−1. Larger values are permitted for 64-bit platforms (except 64-bit Windows, for which large values are truncated to 4GB−1 with a warning).

For additional information about join buffering, see Section 8.2.1.5, “Nested-Loop Join Algorithms”.

Given the aforementioned documentation and your profile, the join_buffer_size is most likely too small. This would trigger mysqld to send the JOIN buffer to a temp table. What makes this worse is that you have six(6) JOIN clauses. That means you have six(6) JOIN buffers.

Trying increasing the join_buffer_size but don't go crazy.

SUGGESTION

Try executing the query dynamically with a larger join_buffer_size.

SET @oldvalue = @@session.join_buffer_size;
SET @newvalue = @oldvalue * 2;
SET SESSION join_buffer_size = @newvalue;
<Run Your Query Here>

Check you EXPLAIN plan and profile and see if that makes a difference.

GIVE IT A TRY !!!

UPDATE 2020-01-15 10:34 EDT

Assuming all indexes for the join exist leaving you with only one JOIN buffer, the last JOIN clause

INNER JOIN reporting.fx_conversion fx ON fx.date = date(ga.created_timestamp) 
                                  AND fx.currency = ga.currency

looks rather nasty because you are using the DATE() function.

That has temp table written all over it. Why ??? That JOIN cannot use an index because of the complex expression. Perhaps one or more JOIN buffers are in use.

  • Why ???: Well, in my defence, this is a problem statement that I've been given to sort out. I can try to remove the date() - I suppose the author thought you can't compare date to timestamp, but I'll try it. – j4nd3r53n Jan 16 at 11:05
  • In reality, you cannot. Once an expression dismisses the use of an index, another JOIN buffer comes into use. The DATE() makes the whole ON expression not to use an index. I once discussed how the internals of a JOIN has to collect the value on an expression. Once functions are introduced in an expression. These internal mechanisms are used. (See my post dba.stackexchange.com/questions/109120/…) – RolandoMySQLDBA Jan 16 at 11:39
  • To be on the safe side, I would not increase join_buffer_size above 1% of RAM. – Rick James Jan 18 at 23:05
0

Two things that virtually force a temp:

GROUP BY <<columns from multiple tables>>
ORDER BY <<columns from multiple tables>>

This

str_to_date(CONCAT(DATE_FORMAT(DATE_SUB(NOW(), INTERVAL 1 WEEK),'%Y-%m-%d'),' 00:00:00'),'%Y-%m-%d %T')

can probably be simplified to

CURDATE() - INTERVAL 1 WEEK

Note: DATE and DATETIME formats can be freely compared. That is, it does not matter which of these you use:

SELECT str_to_date(CONCAT(DATE_FORMAT(DATE_SUB(NOW(),
        INTERVAL 1 WEEK),'%Y-%m-%d'),' 00:00:00'),'%Y-%m-%d %T') AS LongForm,
       CURDATE() - INTERVAL 1 WEEK  AS ShortForm;

+---------------------+------------+
| LongForm            | ShortForm  |
+---------------------+------------+
| 2020-01-11 00:00:00 | 2020-01-11 |
+---------------------+------------+

I suggest you check the math -- doing an aggregate (such as SUM()) after a lot of JOINs may be inflating the values.

We need to see the indexes you have in order to say whether they are optimal.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.