0

We have been facing this issue recently on our MySQL 8 server which has 16 GB RAM and 8 Core Processor. The server runs an application that has the below statistics.

During the time when the data starts inserting into the DB .. during the peak time, the CPU usage goes to 300%.

We have tried many changes but nothing seems to fix it.

The server was working fine till last week. All of a sudden the issue started.

Parameters configured in my.cnf

[mysqld]
innodb_buffer_pool_size = 6442450944
innodb_log_file_size = 512M
sql_mode = STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION
max_connections = 350

enter image description here

>>  MySQLTuner 1.7.17 - Major Hayden <major@mhtx.net>
 >>  Bug reports, feature requests, and downloads at http://mysqltuner.com/
 >>  Run with '--help' for additional options and output filtering

[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[OK] Currently running supported MySQL version 8.0.33-0ubuntu0.22.10.2
[OK] Operating on 64-bit architecture
 
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/log/mysql/error.log exists
[--] Log file: /var/log/mysql/error.log(35K)
[OK] Log file /var/log/mysql/error.log is readable.
[OK] Log file /var/log/mysql/error.log is not empty
[OK] Log file /var/log/mysql/error.log is smaller than 32 Mb
[!!] /var/log/mysql/error.log contains 185 warning(s).
[OK] /var/log/mysql/error.log doesn't contain any error.
[--] 24 start(s) detected in /var/log/mysql/error.log
[--] 1) 2023-05-23T12:55:43.693697Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-0ubuntu0.22.10.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 2) 2023-05-23T12:55:43.693677Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 3) 2023-05-23T11:37:01.249858Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 4) 2023-05-23T11:37:01.249866Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-0ubuntu0.22.10.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 5) 2023-05-23T11:18:19.757007Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 6) 2023-05-23T11:18:19.756922Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-0ubuntu0.22.10.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 7) 2023-05-23T11:17:25.093676Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 8) 2023-05-23T11:17:25.093681Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-0ubuntu0.22.10.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 9) 2023-05-23T11:13:10.652803Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '127.0.0.1' port: 33060, socket: /var/run/mysqld/mysqlx.sock
[--] 10) 2023-05-23T11:13:10.652812Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.33-0ubuntu0.22.10.2'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  (Ubuntu).
[--] 12 shutdown(s) detected in /var/log/mysql/error.log
[--] 1) 2023-05-23T12:55:42.532359Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 2) 2023-05-23T11:37:00.517642Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 3) 2023-05-23T11:17:48.595661Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 4) 2023-05-23T11:17:20.294177Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 5) 2023-05-23T11:13:09.558339Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 6) 2023-05-23T11:09:20.261375Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 7) 2023-05-23T10:50:37.129561Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 8) 2023-05-23T10:45:33.893996Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 9) 2023-05-23T03:38:39.346070Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
[--] 10) 2023-05-23T03:33:59.124231Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.33-0ubuntu0.22.10.2)  (Ubuntu).
 
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +ARCHIVE +BLACKHOLE +CSV -FEDERATED +InnoDB +MEMORY +MRG_MYISAM +MyISAM +PERFORMANCE_SCHEMA 
[--] Data in InnoDB tables: 1.3G (Tables: 71)
[OK] Total fragmented tables: 0
 
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
 
-------- Security Recommendations ------------------------------------------------------------------
[--] Skipped due to unsupported feature for MySQL 8
 
-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION
 
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 43m 58s (440K q [166.797 qps], 144 conn, TX: 399M, RX: 38M)
[--] Reads / Writes: 91% / 9%
[--] Binary logging is enabled (GTID MODE: OFF)
[--] Physical Memory     : 15.6G
[--] Max MySQL memory    : 6.7G
[--] Other process memory: 0B
[--] Total buffers: 6.0G global + 1.9M per thread (350 max threads)
[--] P_S Max memory usage: 72B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 6.1G (39.16% of installed RAM)
[OK] Maximum possible memory usage: 6.7G (42.80% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/440K)
[OK] Highest usage of available connections: 11% (39/350)
[OK] Aborted connections: 0.00%  (0/144)
[!!] name resolution is active : a reverse name resolution is made for each new connection and can reduce performance
[--] Query cache have been removed in MySQL 8
[OK] Sorts requiring temporary tables: 0% (0 temp sorts / 2K sorts)
[!!] Joins performed without indexes: 1536
[OK] Temporary tables created on disk: 0% (0 on disk / 1K total)
[OK] Thread cache hit rate: 72% (39 created / 144 connections)
[OK] Table cache hit rate: 94% (1K open / 1K opened)
[OK] Open file limit used: 0% (2/10K)
[OK] Table locks acquired immediately: 100% (88 immediate / 88 locks)
[OK] Binlog cache memory access: 100.00% (17080 Memory / 17080 Total)
 
-------- Performance schema ------------------------------------------------------------------------
[--] Memory used by P_S: 72B
[--] Sys schema is installed.
 
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
 
-------- MyISAM Metrics ----------------------------------------------------------------------------
[--] MyISAM Metrics are disabled on last MySQL versions.
 
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[OK] InnoDB buffer pool / data size: 6.0G/1.3G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (16.6666666666667 %): 512.0M * 2/6.0G should be equal to 25%
[!!] InnoDB buffer pool instances: 8
[--] Number of InnoDB Buffer Pool Chunk : 48 for 8 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 100.00% (466007045 hits/ 466010295 total)
[!!] InnoDB Write Log efficiency: 43.12% (53408 hits/ 123873 total)
[OK] InnoDB log waits: 0.00% (0 waits / 70465 writes)
 
-------- AriaDB Metrics ----------------------------------------------------------------------------
[--] AriaDB is disabled.
 
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
 
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
 
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
 
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: ROW
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
 
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
    Control warning line(s) into /var/log/mysql/error.log file
    MySQL was started within the last 24 hours - recommendations may be inaccurate
    Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
    Adjust your join queries to always utilize indexes
    Before changing innodb_log_file_size and/or innodb_log_files_in_group read this: https://bit.ly/2TcGgtU
Variables to adjust:
    join_buffer_size (> 256.0K, or always use indexes with JOINs)
    innodb_log_file_size should be (=768M) if possible, so InnoDB total log files size equals to 25% of buffer pool size.
    innodb_buffer_pool_instances(=6)
Improve this question
11
  • Why do you feel you need to do anything about it? On an 8-core server 300% CPU usage means that three out of eight cores are busy, which doesn't look like a problem: CPUs are there to do the computing stuff.
    – mustaccio
    May 23 at 13:59
  • The server stops responding. The apis dont work. And the whole application just dont respond. Need to restart the server. And it will work for another minute and then goes back to halt
    – nXn
    May 23 at 14:00
  • The server was working fine till last week. All of a sudden the issue started , probably any heavy query ?
    – Ergest Basha
    May 23 at 14:02
  • The one action that was done was that a field was made unique . But that caused problem. So we reverted it the next day. Then an update on mysql had happened. MySQL was updated to the latest version . These were the only two things that happened. The load is exactly the same. CPU usage use to be max 50% during peak hours.@ErgestBasha
    – nXn
    May 23 at 14:23
  • @nXn High CPU happens for a lot of reason. Maybe your tables are getting bigger and some queries takes longer. See A Simple Approach to Troubleshooting High CPU in MySQL if it helps
    – Ergest Basha
    May 23 at 14:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.