Skip to main content
lots more
Source Link
Rick James
  • 79.4k
  • 5
  • 51
  • 117

(This is to address the slow queries.)

SELECT  COUNT(*) as `listings2`
    FROM  `ppb_listings`
    WHERE  end_time > NOW()
      AND  deleted = N
      AND  start_time < NOW()

Needs

INDEX(deleted, end_time, start_time),
INDEX(deleted, start_time, end_time)

Neither is really optimal, but they are 'covering', which helps some. The optimizer will dynamically pick which is likely to be better.

SELECT  `value` as `file_path`
    FROM  `ppb_listings_media`
    WHERE  `value` = 'S'
    UNION 
 SELECT  `logo_path` as `file_path`
    FROM  `ppb_categories`
    WHERE  `logo_path` = 'S'
    UNION 
 SELECT  `content` as `file_path`
    FROM  `ppb_advertising`
    WHERE  `content` = 'S'
      AND  `type` = 'S'
    UNION 
 SELECT  `value` as `file_path`
    FROM  `ppb_settings`
    WHERE  `value` = 'S'
      AND  `name` = 'S'
    UNION 
 SELECT  `store_settings` as `file_path`
    FROM  `ppb_users`
    WHERE  `store_settings` LIKE 'S' 

UNION defaults to UNION DISTINCT; change to UNION ALL if practical, since it is faster.

Do you have simple/composite indexes on each table? For example, ppb_settings needs INDEX(value, name) or INDEX(name, value).

SELECT  listings.*
    FROM  ppb_listings AS listings
    WHERE  (closed = N)
      AND  (deleted = N)
      AND  (draft = N)
      AND  (start_time < now())
      AND  (end_time < now())
      AND  (end_time is not null)
    LIMIT  N FOR 
# and  
UPDATE  ppb_listings SET closed = 'S', updated_at = now()
    WHERE  (closed = N)
      AND  (deleted = N)
      AND  (draft = N)
      AND  (start_time < now())
      AND  (end_time > now()
              OR  end_time is null
           ) 

INDEX(closed, deleted, draft, start_time, end_time) (or any of certain rearrangements of such.)

UPDATE  ppb_listings SET deleted = 'S'
    WHERE  (end_time is not null)
      AND  (deleted = N)
      AND  (closed = N)
      AND  (end_time < (now() - interval N day)) 

INDEX(deleted, closed, end_time) -- note that this index and the previous one cannot be combined because end_time is checked in a "range". Also you can probably remove (end_time is not null) since the other test will fail anyway.

SELECT  listings.*
    FROM  ppb_listings AS listings
    WHERE  (counted_at is null
              OR  counted_at < IF(updated_at is null, created_at, updated_at)
           )
    LIMIT  N

Which N do you want? Perhaps you need an ORDER BY? INDEX(counted_at) might help.

SELECT  categories.*
    FROM  ppb_categories AS categories
    WHERE  (enable_auctions = N)
      AND  (user_id is null)
      AND  (parent_id is null)
    ORDER BY  order_id ASC, name ASC 

INDEX(enable_auctions, user_id, parent_id, order_id, name) -- the first 3 in any order; the last 2 must follow and must be in that order.

Those first few probably cover 99% of the activity on the server. Some of my suggestions will help more than others. If you need to discuss any of them, please start another thread, and supply the query, EXPLAIN SELECT ... and SHOW CREATE TABLE.

Those 7 looked like simple cases of "need the right index". If you already have the right index, then perhaps there is no way to improve it. At that point, you need to ask "Does the application really need this query? Or can we change to something else?"


(This is to address the slow queries.)

SELECT  COUNT(*) as `listings2`
    FROM  `ppb_listings`
    WHERE  end_time > NOW()
      AND  deleted = N
      AND  start_time < NOW()

Needs

INDEX(deleted, end_time, start_time),
INDEX(deleted, start_time, end_time)

Neither is really optimal, but they are 'covering', which helps some. The optimizer will dynamically pick which is likely to be better.

SELECT  `value` as `file_path`
    FROM  `ppb_listings_media`
    WHERE  `value` = 'S'
    UNION 
 SELECT  `logo_path` as `file_path`
    FROM  `ppb_categories`
    WHERE  `logo_path` = 'S'
    UNION 
 SELECT  `content` as `file_path`
    FROM  `ppb_advertising`
    WHERE  `content` = 'S'
      AND  `type` = 'S'
    UNION 
 SELECT  `value` as `file_path`
    FROM  `ppb_settings`
    WHERE  `value` = 'S'
      AND  `name` = 'S'
    UNION 
 SELECT  `store_settings` as `file_path`
    FROM  `ppb_users`
    WHERE  `store_settings` LIKE 'S' 

UNION defaults to UNION DISTINCT; change to UNION ALL if practical, since it is faster.

Do you have simple/composite indexes on each table? For example, ppb_settings needs INDEX(value, name) or INDEX(name, value).

SELECT  listings.*
    FROM  ppb_listings AS listings
    WHERE  (closed = N)
      AND  (deleted = N)
      AND  (draft = N)
      AND  (start_time < now())
      AND  (end_time < now())
      AND  (end_time is not null)
    LIMIT  N FOR 
# and  
UPDATE  ppb_listings SET closed = 'S', updated_at = now()
    WHERE  (closed = N)
      AND  (deleted = N)
      AND  (draft = N)
      AND  (start_time < now())
      AND  (end_time > now()
              OR  end_time is null
           ) 

INDEX(closed, deleted, draft, start_time, end_time) (or any of certain rearrangements of such.)

UPDATE  ppb_listings SET deleted = 'S'
    WHERE  (end_time is not null)
      AND  (deleted = N)
      AND  (closed = N)
      AND  (end_time < (now() - interval N day)) 

INDEX(deleted, closed, end_time) -- note that this index and the previous one cannot be combined because end_time is checked in a "range". Also you can probably remove (end_time is not null) since the other test will fail anyway.

SELECT  listings.*
    FROM  ppb_listings AS listings
    WHERE  (counted_at is null
              OR  counted_at < IF(updated_at is null, created_at, updated_at)
           )
    LIMIT  N

Which N do you want? Perhaps you need an ORDER BY? INDEX(counted_at) might help.

SELECT  categories.*
    FROM  ppb_categories AS categories
    WHERE  (enable_auctions = N)
      AND  (user_id is null)
      AND  (parent_id is null)
    ORDER BY  order_id ASC, name ASC 

INDEX(enable_auctions, user_id, parent_id, order_id, name) -- the first 3 in any order; the last 2 must follow and must be in that order.

Those first few probably cover 99% of the activity on the server. Some of my suggestions will help more than others. If you need to discuss any of them, please start another thread, and supply the query, EXPLAIN SELECT ... and SHOW CREATE TABLE.

Those 7 looked like simple cases of "need the right index". If you already have the right index, then perhaps there is no way to improve it. At that point, you need to ask "Does the application really need this query? Or can we change to something else?"

Source Link
Rick James
  • 79.4k
  • 5
  • 51
  • 117

(This is more of a continuation than a new answer. This is a review of the VARIABLES and GLOBAL STATUS.)

Observations: Version: 5.6.28-log
8 GB of RAM
Uptime = 00:51:38; Please rerun SHOW GLOBAL STATUS after several hours.
You are not running on Windows.
Running 64-bit version
It appears that you are running both MyISAM and InnoDB.

The More Important Issues

  • innodb_buffer_pool_size has already been discussed; 1G should be good for now.
  • join_buffer_size, sort_buffer_size should be decreased; see notes below.
  • Created_tmp_disk_tables, Select_scan indicate need for query tuning.
  • thread_cache_size should be no bigger than max_connections; else wasting RAM.

Details and other observations

( innodb_buffer_pool_size / _ram ) = 1024M / 8192M = 12.5% -- % of RAM used for InnoDB buffer_pool

( (key_buffer_size / 0.20 + innodb_buffer_pool_size / 0.70) / _ram ) = (100M / 0.20 + 1024M / 0.70) / 8192M = 24.0% -- Most of available ram should be made available for caching.

( Innodb_buffer_pool_pages_free * 16384 / innodb_buffer_pool_size ) = 59,452 * 16384 / 1024M = 90.7% -- buffer pool free -- buffer_pool_size is bigger than working set; could decrease it

( Innodb_pages_written / Innodb_buffer_pool_write_requests ) = 10,052 / 46262 = 21.7% -- Write requests that had to hit disk -- Check innodb_buffer_pool_size (Probably bogus because of very short Uptime.)

( join_buffer_size ) = 256M -- 0-N per thread. May speed up JOINs (better to fix queries/indexes) (all engines) Used for index scan, range index scan, full table scan, each full JOIN, etc. -- Use the default.

( join_buffer_size / _ram ) = 256M / 8192M = 3.1% -- 0-N per thread. May speed up JOINs (better to fix queries/indexes) (all engines) Used for index scan, range index scan, full table scan, each full JOIN, etc. -- Decrease join_buffer_size to avoid memory pressure. Suggest less than 1% of RAM.

( sort_buffer_size / _ram ) = 256M / 8192M = 3.1% -- Probably should limit to 1% of RAM.

( Key_blocks_used * 1024 / key_buffer_size ) = 331 * 1024 / 100M = 0.32% -- Percent of key_buffer used . High-water-mark. -- Lower key_buffer_size to avoid unnecessary memory usage. (Probably bogus because of very short Uptime.)

( Created_tmp_disk_tables / (Created_tmp_disk_tables + Created_tmp_tables) ) = 8,867 / (8867 + 8999) = 49.6% -- Percent of temp tables that spilled to disk

( tmp_table_size ) = 64M -- Limit on size of MEMORY temp tables used to support a SELECT -- Decrease tmp_table_size to avoid running out of RAM. Perhaps no more than 64M.

( Select_scan / Com_select ) = 15,296 / 22664 = 67.5% -- % of selects doing full table scan. (May be fooled by Stored Routines.) -- Add indexes / optimize queries

( sort_buffer_size ) = 256M -- One per thread, malloced at full size until 5.6.4, so keep low; after that bigger is ok. -- This may be eating into available RAM; recommend no more than 2M.

( binlog_format ) = STATEMENT -- STATEMENT/ROW/MIXED. ROW is preferred; it may become the default.

( Uptime ) = 3,098 = 51m 38s -- How long (in seconds) the server has been running. -- The system has not been up long enough to get reliable suggestions for many of the issues. Fix what you can, then come back with fresh values after the system has been running several hours.

( max_connect_errors ) = 1,000 -- A small protection against hackers. -- Perhaps no more than 200.

( thread_cache_size ) = 16,384 -- How many extra processes to keep around (Not relevant when using thread pooling) (Autosized as of 5.6.8; based on max_connections)

( Threads_created / Connections ) = 263 / 7658 = 3.4% -- Rapidity of process creation (Probably bogus because of very short Uptime.)

( thread_cache_size / max_connections ) = 16,384 / 1000 = 1638.4%

http://mysql.rjweb.org/doc.php/myisam2innodb - http://mysql.rjweb.org/doc.php/memory