Hope someone can provide some ideas...
I have 4 Wordpress websites running on the server. 2 are live sites and 2 are staging sites. Since last week I updated one of Wordpress site to the latest version, the MariaDB server started to be down randomly.
The error message as below: “
kernel: [5855606.614002] Out of memory: Kill process 31213 (mysqld) score 87 or sacrifice child
kernel: [5855606.618908] Killed process 31213 (mysqld) total-vm:1031248kB, anon-rss:351016kB, file-rss:0kB
systemd[1]: mariadb.service: Main process exited, code=killed, status=9/KILL
systemd[1]: mariadb.service: Unit entered failed state.
systemd[1]: mariadb.service: Failed with result ‘signal’.
systemd[1]: mariadb.service: Service hold-off time over, scheduling restart.
systemd[1]: Stopped MariaDB 10.1.46 database server.
systemd[1]: Starting MariaDB 10.1.46 database server…
mysqld[11907]: 2020-12-20 22:26:06 140448597747968 [Note] /usr/sbin/mysqld (mysqld 10.1.46-MariaDB-1~xenial) starting as process 11907 …
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: Using mutexes to ref count buffer pool pages
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: The InnoDB memory heap is disabled
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: Compressed tables use zlib 1.2.8
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: Using Linux native AIO
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: Using SSE crc32 instructions
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] InnoDB: Initializing buffer pool, size = 256.0M
mysqld[11907]: InnoDB: mmap(281149440 bytes) failed; errno 12
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [ERROR] Plugin ‘InnoDB’ init function returned error.
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [ERROR] Plugin ‘InnoDB’ registration as a STORAGE ENGINE failed.
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [Note] Plugin ‘FEEDBACK’ is disabled.
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [ERROR] Unknown/unsupported storage engine: InnoDB
mysqld[11907]: 2020-12-20 22:26:07 140448597747968 [ERROR] Aborting
systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: Failed to start MariaDB 10.1.46 database server.
systemd[1]: mariadb.service: Unit entered failed state.
systemd[1]: mariadb.service: Failed with result ‘exit-code’.”
It seems out of memory and cannot allocate memory for the buffer pool.
I can execute the command "sudo service mysql restart" then the database re-works. But it doesn't solve the issue completely. Maybe a few days later it will be down again.
Add on info - Below are the results of "Show global status"
Aborted_clients 0
Aborted_connects 0
Access_denied_errors 2
Acl_column_grants 0
Acl_database_grants 4
Acl_function_grants 0
Acl_procedure_grants 0
Acl_proxy_users 2
Acl_role_grants 0
Acl_roles 0
Acl_table_grants 0
Acl_users 9
Aria_pagecache_blocks_not_flushed 0
Aria_pagecache_blocks_unused 15706
Aria_pagecache_blocks_used 26
Aria_pagecache_read_requests 515425
Aria_pagecache_reads 58649
Aria_pagecache_write_requests 116877
Aria_pagecache_writes 0
Aria_transaction_log_syncs 0
Binlog_commits 297194
Binlog_group_commits 297194
Binlog_group_commit_trigger_count 0
Binlog_group_commit_trigger_lock_wait 0
Binlog_group_commit_trigger_timeout 0
Binlog_snapshot_file mariadb-bin.000540
Binlog_snapshot_position 24383614
Binlog_bytes_written 234313344
Binlog_cache_disk_use 221
Binlog_cache_use 297187
Binlog_stmt_cache_disk_use 0
Binlog_stmt_cache_use 7
Busy_time 0
Bytes_received 549059677
Bytes_sent 20452078223
Com_admin_commands 1
Com_alter_db 0
Com_alter_db_upgrade 0
Com_alter_event 0
Com_alter_function 0
Com_alter_procedure 0
Com_alter_server 0
Com_alter_table 2
Com_alter_tablespace 0
Com_analyze 0
Com_assign_to_keycache 0
Com_begin 0
Com_binlog 0
Com_call_procedure 0
Com_change_db 16542
Com_change_master 0
Com_check 0
Com_checksum 0
Com_commit 3
Com_compound_sql 0
Com_create_db 0
Com_create_event 0
Com_create_function 0
Com_create_index 0
Com_create_procedure 0
Com_create_role 0
Com_create_server 0
Com_create_table 5407
Com_create_temporary_table 0
Com_create_trigger 0
Com_create_udf 0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 0
Com_delete 11736
Com_delete_multi 34
Com_do 0
Com_drop_db 0
Com_drop_event 0
Com_drop_function 0
Com_drop_index 0
Com_drop_procedure 0
Com_drop_role 0
Com_drop_server 0
Com_drop_table 0
Com_drop_temporary_table 0
Com_drop_trigger 0
Com_drop_user 0
Com_drop_view 0
Com_empty_query 0
Com_execute_sql 0
Com_flush 12
Com_get_diagnostics 0
Com_grant 0
Com_grant_role 0
Com_ha_close 0
Com_ha_open 0
Com_ha_read 0
Com_help 0
Com_insert 236211
Com_insert_select 8
Com_install_plugin 0
Com_kill 0
Com_load 0
Com_lock_tables 0
Com_optimize 0
Com_preload_keys 0
Com_prepare_sql 0
Com_purge 0
Com_purge_before_date 0
Com_release_savepoint 0
Com_rename_table 0
Com_rename_user 0
Com_repair 0
Com_replace 113
Com_replace_select 0
Com_reset 0
Com_resignal 0
Com_revoke 0
Com_revoke_all 0
Com_revoke_role 0
Com_rollback 0
Com_rollback_to_savepoint 0
Com_savepoint 0
Com_select 2182647
Com_set_option 49707
Com_show_authors 0
Com_show_binlog_events 0
Com_show_binlogs 1
Com_show_charsets 0
Com_show_collations 0
Com_show_contributors 0
Com_show_create_db 0
Com_show_create_event 0
Com_show_create_func 0
Com_show_create_proc 0
Com_show_create_table 0
Com_show_create_trigger 0
Com_show_databases 0
Com_show_engine_logs 0
Com_show_engine_mutex 0
Com_show_engine_status 0
Com_show_errors 0
Com_show_events 0
Com_show_explain 0
Com_show_fields 52353
Com_show_function_status 0
Com_show_generic 0
Com_show_grants 1
Com_show_keys 59
Com_show_master_status 0
Com_show_open_tables 0
Com_show_plugins 0
Com_show_privileges 0
Com_show_procedure_status 0
Com_show_processlist 0
Com_show_profile 0
Com_show_profiles 0
Com_show_relaylog_events 0
Com_show_slave_hosts 0
Com_show_slave_status 0
Com_show_status 3
Com_show_storage_engines 0
Com_show_table_status 0
Com_show_tables 45
Com_show_triggers 0
Com_show_variables 1528
Com_show_warnings 0
Com_shutdown 0
Com_signal 0
Com_start_all_slaves 0
Com_start_slave 0
Com_stmt_close 1508
Com_stmt_execute 1508
Com_stmt_fetch 0
Com_stmt_prepare 1508
Com_stmt_reprepare 0
Com_stmt_reset 0
Com_stmt_send_long_data 1508
Com_stop_all_slaves 0
Com_stop_slave 0
Com_truncate 59
Com_uninstall_plugin 0
Com_unlock_tables 0
Com_update 49139
Com_update_multi 0
Com_xa_commit 0
Com_xa_end 0
Com_xa_prepare 0
Com_xa_recover 0
Com_xa_rollback 0
Com_xa_start 0
Compression OFF
Connection_errors_accept 0
Connection_errors_internal 0
Connection_errors_max_connections 0
Connection_errors_peer_address 0
Connection_errors_select 0
Connection_errors_tcpwrap 0
Connections 16623
Cpu_time 0
Created_tmp_disk_tables 58860
Created_tmp_files 61
Created_tmp_tables 66826
Delayed_errors 0
Delayed_insert_threads 0
Delayed_writes 0
Delete_scan 190
Empty_queries 1054229
Executed_events 0
Executed_triggers 0
Feature_delay_key_write 0
Feature_dynamic_columns 0
Feature_fulltext 0
Feature_gis 0
Feature_locale 28
Feature_subquery 11932
Feature_timezone 0
Feature_trigger 0
Feature_xml 0
Flush_commands 1
Handler_commit 1431481
Handler_delete 15451
Handler_discover 0
Handler_external_lock 0
Handler_icp_attempts 30792448
Handler_icp_match 30792107
Handler_mrr_init 0
Handler_mrr_key_refills 0
Handler_mrr_rowid_refills 0
Handler_prepare 572504
Handler_read_first 4738
Handler_read_key 1143192
Handler_read_last 632
Handler_read_next 42445143
Handler_read_prev 16431
Handler_read_retry 0
Handler_read_rnd 708706
Handler_read_rnd_deleted 18
Handler_read_rnd_next 10019868
Handler_rollback 47
Handler_savepoint 0
Handler_savepoint_rollback 0
Handler_tmp_update 4045
Handler_tmp_write 6615334
Handler_update 176761
Handler_write 337672
Innodb_available_undo_logs 128
Innodb_background_log_sync 273531
Innodb_buffer_pool_bytes_data 155009024
Innodb_buffer_pool_bytes_dirty 0
Innodb_buffer_pool_dump_status Dumping buffer pool(s) not yet started
Innodb_buffer_pool_load_status Loading buffer pool(s) not yet started
Innodb_buffer_pool_pages_data 9461
Innodb_buffer_pool_pages_dirty 0
Innodb_buffer_pool_pages_flushed 252710
Innodb_buffer_pool_pages_free 6542
Innodb_buffer_pool_pages_lru_flushed 0
Innodb_buffer_pool_pages_made_not_young 0
Innodb_buffer_pool_pages_made_young 29
Innodb_buffer_pool_pages_misc 380
Innodb_buffer_pool_pages_old 3511
Innodb_buffer_pool_pages_total 16383
Innodb_buffer_pool_read_ahead 241
Innodb_buffer_pool_read_ahead_evicted 0
Innodb_buffer_pool_read_ahead_rnd 0
Innodb_buffer_pool_read_requests 127140642
Innodb_buffer_pool_reads 6290
Innodb_buffer_pool_wait_free 0
Innodb_buffer_pool_write_requests 3250469
Innodb_checkpoint_age 0
Innodb_checkpoint_max_age 80826164
Innodb_data_fsyncs 330303
Innodb_data_pending_fsyncs 0
Innodb_data_pending_reads 0
Innodb_data_pending_writes 0
Innodb_data_read 109564416
Innodb_data_reads 7904
Innodb_data_writes 549421
Innodb_data_written 8816224256
Innodb_dblwr_pages_written 252710
Innodb_dblwr_writes 9079
Innodb_deadlocks 0
Innodb_have_atomic_builtins ON
Innodb_history_list_length 5
Innodb_ibuf_discarded_delete_marks 0
Innodb_ibuf_discarded_deletes 0
Innodb_ibuf_discarded_inserts 0
Innodb_ibuf_free_list 23
Innodb_ibuf_merged_delete_marks 665
Innodb_ibuf_merged_deletes 97
Innodb_ibuf_merged_inserts 71
Innodb_ibuf_merges 70
Innodb_ibuf_segment_size 25
Innodb_ibuf_size 1
Innodb_log_waits 0
Innodb_log_write_requests 853062
Innodb_log_writes 279156
Innodb_lsn_current 101526933039
Innodb_lsn_flushed 101526933039
Innodb_lsn_last_checkpoint 101526933039
Innodb_master_thread_active_loops 11800
Innodb_master_thread_idle_loops 261798
Innodb_max_trx_id 289624446
Innodb_mem_adaptive_hash 10655952
Innodb_mem_dictionary 2340344
Innodb_mem_total 281149440
Innodb_mutex_os_waits 42758
Innodb_mutex_spin_rounds 1516853
Innodb_mutex_spin_waits 188626
Innodb_oldest_view_low_limit_trx_id 0
Innodb_os_log_fsyncs 285934
Innodb_os_log_pending_fsyncs 0
Innodb_os_log_pending_writes 0
Innodb_os_log_written 529884160
Innodb_page_size 16384
Innodb_pages_created 2880
Innodb_pages_read 6581
Innodb_pages0_read 1211
Innodb_pages_written 252710
Innodb_purge_trx_id 289624441
Innodb_purge_undo_no 0
Innodb_read_views_memory 200
Innodb_row_lock_current_waits 0
Innodb_row_lock_time 26
Innodb_row_lock_time_avg 1
Innodb_row_lock_time_max 4
Innodb_row_lock_waits 15
Innodb_rows_deleted 15451
Innodb_rows_inserted 176801
Innodb_rows_read 51097494
Innodb_rows_updated 176760
Innodb_system_rows_deleted 0
Innodb_system_rows_inserted 0
Innodb_system_rows_read 0
Innodb_system_rows_updated 0
Innodb_s_lock_os_waits 11270
Innodb_s_lock_spin_rounds 343397
Innodb_s_lock_spin_waits 11714
Innodb_truncated_status_writes 0
Innodb_x_lock_os_waits 828
Innodb_x_lock_spin_rounds 26073
Innodb_x_lock_spin_waits 697
Innodb_page_compression_saved 0
Innodb_page_compression_trim_sect512 0
Innodb_page_compression_trim_sect1024 0
Innodb_page_compression_trim_sect2048 0
Innodb_page_compression_trim_sect4096 0
Innodb_page_compression_trim_sect8192 0
Innodb_page_compression_trim_sect16384 0
Innodb_page_compression_trim_sect32768 0
Innodb_num_index_pages_written 63085
Innodb_num_non_index_pages_written 485604
Innodb_num_pages_page_compressed 0
Innodb_num_page_compressed_trim_op 0
Innodb_num_page_compressed_trim_op_saved 0
Innodb_num_pages_page_decompressed 0
Innodb_num_pages_page_compression_error 0
Innodb_num_pages_encrypted 0
Innodb_num_pages_decrypted 0
Innodb_have_lz4 ON
Innodb_have_lzo OFF
Innodb_have_lzma OFF
Innodb_have_bzip2 OFF
Innodb_have_snappy OFF
Innodb_defragment_compression_failures 0
Innodb_defragment_failures 0
Innodb_defragment_count 0
Innodb_onlineddl_rowlog_rows 0
Innodb_onlineddl_rowlog_pct_used 0
Innodb_onlineddl_pct_progress 0
Innodb_secondary_index_triggered_cluster_reads 35477260
Innodb_secondary_index_triggered_cluster_reads_avo... 0
Innodb_buffered_aio_submitted 241
Innodb_encryption_rotation_pages_read_from_cache 0
Innodb_encryption_rotation_pages_read_from_disk 0
Innodb_encryption_rotation_pages_modified 0
Innodb_encryption_rotation_pages_flushed 0
Innodb_encryption_rotation_estimated_iops 0
Innodb_encryption_key_rotation_list_length 0
Innodb_encryption_n_merge_blocks_encrypted 0
Innodb_encryption_n_merge_blocks_decrypted 0
Innodb_encryption_n_rowlog_blocks_encrypted 0
Innodb_encryption_n_rowlog_blocks_decrypted 0
Innodb_scrub_background_page_reorganizations 0
Innodb_scrub_background_page_splits 0
Innodb_scrub_background_page_split_failures_underf... 0
Innodb_scrub_background_page_split_failures_out_of... 0
Innodb_scrub_background_page_split_failures_missin... 0
Innodb_scrub_background_page_split_failures_unknow... 0
Innodb_encryption_num_key_requests 0
Key_blocks_not_flushed 0
Key_blocks_unused 107161
Key_blocks_used 2
Key_blocks_warm 0
Key_read_requests 2
Key_reads 2
Key_write_requests 2
Key_writes 2
Last_query_cost 0
Master_gtid_wait_count 0
Master_gtid_wait_time 0
Master_gtid_wait_timeouts 0
Max_statement_time_exceeded 0
Max_used_connections 9
Memory_used 348877656
Not_flushed_delayed_rows 0
Open_files 10
Open_streams 0
Open_table_definitions 400
Open_tables 400
Opened_files 236650
Opened_plugin_libraries 0
Opened_table_definitions 490
Opened_tables 595
Opened_views 0
Performance_schema_accounts_lost 0
Performance_schema_cond_classes_lost 0
Performance_schema_cond_instances_lost 0
Performance_schema_digest_lost 0
Performance_schema_file_classes_lost 0
Performance_schema_file_handles_lost 0
Performance_schema_file_instances_lost 0
Performance_schema_hosts_lost 0
Performance_schema_locker_lost 0
Performance_schema_mutex_classes_lost 0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost 0
Performance_schema_rwlock_instances_lost 0
Performance_schema_session_connect_attrs_lost 0
Performance_schema_socket_classes_lost 0
Performance_schema_socket_instances_lost 0
Performance_schema_stage_classes_lost 0
Performance_schema_statement_classes_lost 0
Performance_schema_table_handles_lost 0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost 0
Performance_schema_thread_instances_lost 0
Performance_schema_users_lost 0
Prepared_stmt_count 0
Qcache_free_blocks 11826
Qcache_free_memory 33876616
Qcache_hits 1312598
Qcache_inserts 704079
Qcache_lowmem_prunes 22628
Qcache_not_cached 185838
Qcache_queries_in_cache 19580
Qcache_total_blocks 51437
Queries 2626873
Questions 2623857
Rows_read 53084304
Rows_sent 44080973
Rows_tmp_read 459267
Rpl_status AUTH_MASTER
Select_full_join 36
Select_full_range_join 0
Select_range 26234
Select_range_check 0
Select_scan 95723
Slave_connections 0
Slave_heartbeat_period 0
Slave_open_temp_tables 0
Slave_received_heartbeats 0
Slave_retried_transactions 0
Slave_running OFF
Slave_skipped_errors 0
Slaves_connected 0
Slaves_running 0
Slow_launch_threads 0
Slow_queries 0
Sort_merge_passes 0
Sort_priority_queue_sorts 22368
Sort_range 100258
Sort_rows 730332
Sort_scan 14387
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher
Ssl_cipher_list
Ssl_client_connects 0
Ssl_connect_renegotiates 0
Ssl_ctx_verify_depth 0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts 0
Ssl_finished_connects 0
Ssl_server_not_after
Ssl_server_not_before
Ssl_session_cache_hits 0
Ssl_session_cache_misses 0
Ssl_session_cache_mode NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size 0
Ssl_session_cache_timeouts 0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries 0
Ssl_verify_depth 0
Ssl_verify_mode 0
Ssl_version
Subquery_cache_hit 0
Subquery_cache_miss 181
Syncs 19
Table_locks_immediate 1170023
Table_locks_waited 0
Tc_log_max_pages_used 0
Tc_log_page_size 0
Tc_log_page_waits 0
Threadpool_idle_threads 0
Threadpool_threads 0
Threads_cached 7
Threads_connected 2
Threads_created 9
Threads_running 1
Update_scan 7
Uptime 273900
Uptime_since_flush_status 273900
wsrep_cluster_conf_id 18446744073709551615
wsrep_cluster_size 0
wsrep_cluster_state_uuid
wsrep_cluster_status Disconnected
wsrep_connected OFF
wsrep_local_bf_aborts 0
wsrep_local_index 18446744073709551615
wsrep_provider_name
wsrep_provider_vendor
wsrep_provider_version
wsrep_ready OFF
wsrep_thread_count 0
Anyone know what’s possible reason to cause the issue and how to fix it?
Many thanks!
