Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

We have a server running LAMP stack on CentOS release 5.6 (Final) MySQL is having high CPU usage constantly which causes server to lag as shown top - 13:50:47 up 80 days, 13:53, 1 user, load average: 4.02, 3.44, 3.19 Tasks: 129 total, 2 running, 127 sleeping, 0 stopped, 0 zombie Cpu0 : 24.7%us, 19.8%sy, 0.2%ni, 54.5%id, 0.7%wa, 0.0%hi, 0.1%si, 0.0%st Cpu1 : 23.0%us, 9.4%sy, 0.1%ni, 67.0%id, 0.3%wa, 0.0%hi, 0.2%si, 0.0%st Mem: 4034120k total, 3317400k used, 716720k free, 124324k buffers Swap: 8385920k total, 9788k used, 8376132k free, 2004692k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 26266 mysql 15 0 2472m 464m 4448 S 158.6 11.8 109:00.59 mysqld 30276 apache 16 0 310m 16m 3724 S 12.1 0.4 0:00.85 httpd 30205 apache 16 0 309m 17m 3860 S 8.6 0.4 0:01.81 httpd

The server has 4GB RAM, tables are stored in MyISAM. Database is about 6665MB. It has a replicated database (readonly) on another server

MySQL version: +----------------+ | 5.1.57-ius-log | +----------------+ Here's our my.cnf (IP information Xed out). [mysqld] # General datadir = /home/mysql tmpdir = /var/lib/mysqltmp socket = /var/lib/mysql/mysql.sock skip-external-locking = 1 skip-name-resolve open-files-limit = 20000 # Cache thread-cache-size = 16 table-open-cache = 2048 table-definition-cache = 512 query-cache-size = 32M query-cache-limit = 1M # Per-thread Buffers sort-buffer-size = 1M read-buffer-size = 1M read-rnd-buffer-size = 8M join-buffer-size = 1M # Temp Tables tmp-table-size = 64M max-heap-table-size = 64M # Networking back-log = 100 max-connections = 200 max-connect-errors = 10000 max-allowed-packet = 16M interactive-timeout = 600 wait-timeout = 600 #Storage Engines innodb = FORCE # MyISAM key-buffer-size = 64M myisam-sort-buffer-size = 128M # InnoDB innodb-buffer-pool-size = 16M innodb_log_files_in_group = 2 innodb-log-buffer-size = 4M # Replication server-id = 1 bind-address = XX.XX.XXX.XXX log-bin = /var/lib/mysqllogs/bin-log binlog-do-db = rewards1 binlog-ignore-db = mysql relay-log-index=master-relay-bin.index relay-log = /var/lib/mysqllogs/relay-log #Logging slow-query-log = 1 slow-query-log-file = /var/log/slow-log #MysqlTuner skip-innodb max_connections = 200 wait_timeout = 500 interactive_timeout = 500 query_cache_size = 64M join_buffer_size = 5M tmp_table_size = 256M max_heap_table_size = 256M table_cache = 4096 key_buffer_size = 2048M [mysqld_safe] log-error = /var/log/mysqld.log

We've seen similar problem as described in MySQL high CPU usage (MyISAM table indexes) which I've adjust the key_buffer_size to 2048M so it's greater than MyISAM index mysql> sELECT SUM(index_length) ndxsize FROM information_schema.tables WHERE engine='MyISAM'; +------------+ | ndxsize | +------------+ | 1876895744 | +------------+ MySQL continues to have high %CPU, our system admin suggested that we add more RAM to the box, will this solve the problem?

Any helps/suggestions are appreciated!!

Thank you!!!

Edited for show status, it seems that show status only shows session status, changed it to run global status mysql> show global status; +--------------------------------+--------------+ | Variable_name | Value | +--------------------------------+--------------+ | Aborted_clients | 15 | | Aborted_connects | 101 | | Binlog_cache_disk_use | 0 | | Binlog_cache_use | 0 | | Bytes_received | 4820268802 | | Bytes_sent | 104046088856 | | Com_admin_commands | 2 | | Com_assign_to_keycache | 0 | | Com_alter_db | 0 | | Com_alter_db_upgrade | 0 | | Com_alter_event | 0 | | Com_alter_function | 0 | | Com_alter_procedure | 0 | | Com_alter_server | 0 | | Com_alter_table | 0 | | Com_alter_tablespace | 0 | | Com_analyze | 0 | | Com_backup_table | 0 | | Com_begin | 0 | | Com_binlog | 0 | | Com_call_procedure | 0 | | Com_change_db | 1376751 | | Com_change_master | 0 | | Com_check | 0 | | Com_checksum | 2 | | Com_commit | 0 | | Com_create_db | 0 | | Com_create_event | 0 | | Com_create_function | 0 | | Com_create_index | 1 | | Com_create_procedure | 0 | | Com_create_server | 0 | | Com_create_table | 133 | | Com_create_trigger | 0 | | Com_create_udf | 0 | | Com_create_user | 0 | | Com_create_view | 0 | | Com_dealloc_sql | 0 | | Com_delete | 31518 | | Com_delete_multi | 0 | | Com_do | 0 | | Com_drop_db | 0 | | Com_drop_event | 0 | | Com_drop_function | 0 | | Com_drop_index | 0 | | Com_drop_procedure | 0 | | Com_drop_server | 0 | | Com_drop_table | 133 | | Com_drop_trigger | 0 | | Com_drop_user | 0 | | Com_drop_view | 0 | | Com_empty_query | 0 | | Com_execute_sql | 0 | | Com_flush | 0 | | Com_grant | 0 | | Com_ha_close | 0 | | Com_ha_open | 0 | | Com_ha_read | 0 | | Com_help | 0 | | Com_insert | 140425 | | Com_insert_select | 137 | | Com_install_plugin | 0 | | Com_kill | 0 | | Com_load | 0 | | Com_load_master_data | 0 | | Com_load_master_table | 0 | | Com_lock_tables | 0 | | Com_optimize | 0 | | Com_preload_keys | 0 | | Com_prepare_sql | 0 | | Com_purge | 0 | | Com_purge_before_date | 0 | | Com_release_savepoint | 0 | | Com_rename_table | 0 | | Com_rename_user | 0 | | Com_repair | 0 | | Com_replace | 5805 | | Com_replace_select | 0 | | Com_reset | 0 | | Com_restore_table | 0 | | Com_revoke | 0 | | Com_revoke_all | 0 | | Com_rollback | 3391 | | Com_rollback_to_savepoint | 0 | | Com_savepoint | 0 | | Com_select | 7088783 | | Com_set_option | 1226039 | | Com_show_authors | 0 | | Com_show_binlog_events | 0 | | Com_show_binlogs | 0 | | Com_show_charsets | 0 | | Com_show_collations | 0 | | Com_show_column_types | 0 | | Com_show_contributors | 0 | | Com_show_create_db | 0 | | Com_show_create_event | 0 | | Com_show_create_func | 0 | | Com_show_create_proc | 0 | | Com_show_create_table | 2 | | Com_show_create_trigger | 0 | | Com_show_databases | 24 | | Com_show_engine_logs | 0 | | Com_show_engine_mutex | 0 | | Com_show_engine_status | 0 | | Com_show_events | 0 | | Com_show_errors | 0 | | Com_show_fields | 17299 | | Com_show_function_status | 0 | | Com_show_grants | 0 | | Com_show_keys | 0 | | Com_show_master_status | 0 | | Com_show_new_master | 0 | | Com_show_open_tables | 0 | | Com_show_plugins | 0 | | Com_show_privileges | 0 | | Com_show_procedure_status | 0 | | Com_show_processlist | 131 | | Com_show_profile | 0 | | Com_show_profiles | 0 | | Com_show_slave_hosts | 0 | | Com_show_slave_status | 0 | | Com_show_status | 10990 | | Com_show_storage_engines | 0 | | Com_show_table_status | 13 | | Com_show_tables | 23 | | Com_show_triggers | 0 | | Com_show_variables | 3400 | | Com_show_warnings | 0 | | Com_slave_start | 0 | | Com_slave_stop | 0 | | Com_stmt_close | 0 | | Com_stmt_execute | 0 | | Com_stmt_fetch | 0 | | Com_stmt_prepare | 0 | | Com_stmt_reprepare | 0 | | Com_stmt_reset | 0 | | Com_stmt_send_long_data | 0 | | Com_truncate | 133 | | Com_uninstall_plugin | 0 | | Com_unlock_tables | 0 | | Com_update | 713677 | | Com_update_multi | 133 | | Com_xa_commit | 0 | | Com_xa_end | 0 | | Com_xa_prepare | 0 | | Com_xa_recover | 0 | | Com_xa_rollback | 0 | | Com_xa_start | 0 | | Compression | OFF | | Connections | 1381196 | | Created_tmp_disk_tables | 500962 | | Created_tmp_files | 17174 | | Created_tmp_tables | 1323921 | | Delayed_errors | 0 | | Delayed_insert_threads | 0 | | Delayed_writes | 0 | | Flush_commands | 1 | | Handler_commit | 0 | | Handler_delete | 38013 | | Handler_discover | 0 | | Handler_prepare | 0 | | Handler_read_first | 326121 | | Handler_read_key | 141753461 | | Handler_read_next | 2628857102 | | Handler_read_prev | 65624374428 | | Handler_read_rnd | 55126444 | | Handler_read_rnd_next | 9990315545 | | Handler_rollback | 0 | | Handler_savepoint | 0 | | Handler_savepoint_rollback | 0 | | Handler_update | 10330960 | | Handler_write | 22915553 | | Key_blocks_not_flushed | 0 | | Key_blocks_unused | 1424358 | | Key_blocks_used | 297815 | | Key_read_requests | 3988011306 | | Key_reads | 294476 | | Key_write_requests | 9047373 | | Key_writes | 439479 | | Last_query_cost | 0.000000 | | Max_used_connections | 81 | | Not_flushed_delayed_rows | 0 | | Open_files | 1623 | | Open_streams | 0 | | Open_table_definitions | 531 | | Open_tables | 1093 | | Opened_files | 2242361 | | Opened_table_definitions | 1076 | | Opened_tables | 1655 | | Prepared_stmt_count | 0 | | Qcache_free_blocks | 10115 | | Qcache_free_memory | 36606896 | | Qcache_hits | 6401379 | | Qcache_inserts | 6086477 | | Qcache_lowmem_prunes | 539333 | | Qcache_not_cached | 1001986 | | Qcache_queries_in_cache | 17468 | | Qcache_total_blocks | 45584 | | Queries | 18400635 | | Questions | 18400635 | | Rpl_status | NULL | | Select_full_join | 29962 | | Select_full_range_join | 0 | | Select_range | 2141059 | | Select_range_check | 0 | | Select_scan | 2084788 | | Slave_open_temp_tables | 0 | | Slave_retried_transactions | 0 | | Slave_running | OFF | | Slow_launch_threads | 4 | | Slow_queries | 4 | | Sort_merge_passes | 8587 | | Sort_range | 1924836 | | Sort_rows | 117048520 | | Sort_scan | 1366104 | | Ssl_accept_renegotiates | 0 | | Ssl_accepts | 0 | | Ssl_callback_cache_hits | 0 | | Ssl_cipher | | | Ssl_cipher_list | | | Ssl_client_connects | 0 | | Ssl_connect_renegotiates | 0 | | Ssl_ctx_verify_depth | 0 | | Ssl_ctx_verify_mode | 0 | | Ssl_default_timeout | 0 | | Ssl_finished_accepts | 0 | | Ssl_finished_connects | 0 | | Ssl_session_cache_hits | 0 | | Ssl_session_cache_misses | 0 | | Ssl_session_cache_mode | NONE | | Ssl_session_cache_overflows | 0 | | Ssl_session_cache_size | 0 | | Ssl_session_cache_timeouts | 0 | | Ssl_sessions_reused | 0 | | Ssl_used_session_cache_entries | 0 | | Ssl_verify_depth | 0 | | Ssl_verify_mode | 0 | | Ssl_version | | | Table_locks_immediate | 16998029 | | Table_locks_waited | 236687 | | Tc_log_max_pages_used | 0 | | Tc_log_page_size | 0 | | Tc_log_page_waits | 0 | | Threads_cached | 10 | | Threads_connected | 7 | | Threads_created | 5861 | | Threads_running | 6 | | Uptime | 250234 | | Uptime_since_flush_status | 250234 | +--------------------------------+--------------+ 249 rows in set (0.00 sec)

Show processlist during huge lag spike: mysql> show processlist; +---------+--------------+-----------+----------+---------+------+--------------+-------+ | Id | User | Host | db | Command | Time | State | Info | +---------+--------------+-----------+----------+---------+------+--------------+------------------------------------------------------------------------------------------------------+ | 1421036 | rewards_user | localhost | rewards1 | Sleep | 478 | | NULL | | 1421054 | rewards_user | localhost | rewards1 | Sleep | 478 | | NULL | | 1421084 | rewards_user | localhost | rewards1 | Sleep | 474 | | NULL | | 1421188 | rewards_user | localhost | rewards1 | Sleep | 456 | | NULL | | 1421296 | rewards_user | localhost | rewards1 | Sleep | 435 | | NULL | | 1421298 | rewards_user | localhost | rewards1 | Sleep | 435 | | NULL | | 1421334 | rewards_user | localhost | rewards1 | Sleep | 430 | | NULL | | 1421335 | rewards_user | localhost | rewards1 | Sleep | 430 | | NULL | | 1421342 | rewards_user | localhost | rewards1 | Sleep | 426 | | NULL | | 1421405 | rewards_user | localhost | rewards1 | Sleep | 415 | | NULL | | 1421459 | rewards_user | localhost | rewards1 | Sleep | 409 | | NULL | | 1421494 | rewards_user | localhost | rewards1 | Sleep | 404 | | NULL | | 1421513 | rewards_user | localhost | rewards1 | Sleep | 399 | | NULL |

after mysql recovers

mysql> show processlist; +---------+--------------+-----------+----------+---------+------+-------+------------------+ | Id | User | Host | db | Command | Time | State | Info | +---------+--------------+-----------+----------+---------+------+-------+------------------+ | 1423101 | rewards_user | localhost | rewards1 | Query | 0 | NULL | show processlist | | 1427422 | rewards_user | localhost | rewards1 | Sleep | 0 | | NULL | +---------+--------------+-----------+----------+---------+------+-------+------------------+ 2 rows in set (0.00 sec)

Thanks again!! appreciated :)

share|improve this question
    
Could you show the output of SHOW STATUS command from mysql? –  Tero Kilkanen Mar 22 at 10:05
    
Added the result for SHOW STATUS. Thanks! –  wayne Mar 22 at 17:16
    
It looks like this SHOW STATUS was taken just after server restart? The server should be running for a while and serving normal requests before SHOW STATUS has good information. –  Tero Kilkanen Mar 22 at 17:19
    
I might have restarted mysql yesterday due to the heavy lag. How long should I wait till next mysql status measure? Thanks! –  wayne Mar 23 at 0:13
    
A couple of hours shoule be enough. –  Tero Kilkanen Mar 23 at 9:35

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.