Help needed in tuning mysql version 5.5 [closed]

Question

I just migrated a 32 bits MySQL 5.5 to a 64 bit version of MySQL 5.5. I would like to upgrade it to 8.0. But my company has some ancient instruments. I can not do it right now before I check the compatibility. The old machine is using the 32 bits of MySQL version which only allows it to use 4GB of RAM. Right now, with the 64 bit version, I already made some changes like increase the innodb_buffer_pool_size to 196 GB. And for searching non-indexed data, it speeds up 2 times. But some of my colleagues says some queries are slower. I wonder whether you guys can help me to take a look. I tried to use some tuning scripts but some of them shows this version is unsupported.

Basic Machine information: 48 cores CPU with a 378 G of RAM, 4TB SSD in Raid 10.

The result for show global status:

Aborted_clients 1802
Aborted_connects    24
Binlog_cache_disk_use   28
Binlog_cache_use    19317
Binlog_stmt_cache_disk_use  0
Binlog_stmt_cache_use   0
Bytes_received  202761065
Bytes_sent  502634219
Com_admin_commands  164680
Com_assign_to_keycache  0
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function  0
Com_alter_procedure 0
Com_alter_server    0
Com_alter_table 3
Com_alter_tablespace    0
Com_analyze 0
Com_begin   0
Com_binlog  0
Com_call_procedure  0
Com_change_db   3647
Com_change_master   0
Com_check   0
Com_checksum    0
Com_commit  3760
Com_create_db   1
Com_create_event    0
Com_create_function 21
Com_create_index    0
Com_create_procedure    0
Com_create_server   0
Com_create_table    17
Com_create_trigger  2
Com_create_udf  0
Com_create_user 0
Com_create_view 13
Com_dealloc_sql 0
Com_delete  10
Com_delete_multi    0
Com_do  0
Com_drop_db 0
Com_drop_event  0
Com_drop_function   21
Com_drop_index  0
Com_drop_procedure  0
Com_drop_server 0
Com_drop_table  15
Com_drop_trigger    2
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_execute_sql 0
Com_flush   1
Com_grant   0
Com_ha_close    0
Com_ha_open 0
Com_ha_read 0
Com_help    0
Com_insert  93958
Com_insert_select   0
Com_install_plugin  0
Com_kill    0
Com_load    0
Com_lock_tables 0
Com_optimize    0
Com_preload_keys    0
Com_prepare_sql 0
Com_purge   0
Com_purge_before_date   0
Com_release_savepoint   0
Com_rename_table    0
Com_rename_user 0
Com_repair  0
Com_replace 6
Com_replace_select  0
Com_reset   0
Com_resignal    0
Com_revoke  0
Com_revoke_all  0
Com_rollback    140484
Com_rollback_to_savepoint   0
Com_savepoint   0
Com_select  175771
Com_set_option  339993
Com_signal  0
Com_show_authors    0
Com_show_binlog_events  0
Com_show_binlogs    0
Com_show_charsets   19
Com_show_collations 111
Com_show_contributors   0
Com_show_create_db  0
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_proc    0
Com_show_create_table   12996
Com_show_create_trigger 0
Com_show_databases  18
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  0
Com_show_events 0
Com_show_errors 0
Com_show_fields 48
Com_show_function_status    6
Com_show_grants 2
Com_show_keys   25
Com_show_master_status  0
Com_show_open_tables    0
Com_show_plugins    0
Com_show_privileges 0
Com_show_procedure_status   6
Com_show_processlist    7
Com_show_profile    0
Com_show_profiles   0
Com_show_relaylog_events    0
Com_show_slave_hosts    0
Com_show_slave_status   0
Com_show_status 43
Com_show_storage_engines    0
Com_show_table_status   0
Com_show_tables 53
Com_show_triggers   5
Com_show_variables  1003
Com_show_warnings   0
Com_slave_start 0
Com_slave_stop  0
Com_stmt_close  0
Com_stmt_execute    0
Com_stmt_fetch  0
Com_stmt_prepare    0
Com_stmt_reprepare  0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_truncate    0
Com_uninstall_plugin    0
Com_unlock_tables   0
Com_update  1751
Com_update_multi    0
Com_xa_commit   0
Com_xa_end  0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression OFF
Connections 72272
Created_tmp_disk_tables 947
Created_tmp_files   60
Created_tmp_tables  2356
Delayed_errors  0
Delayed_insert_threads  0
Delayed_writes  0
Flush_commands  1
Handler_commit  295935
Handler_delete  4487
Handler_discover    0
Handler_prepare 196894
Handler_read_first  21517
Handler_read_key    13718811
Handler_read_last   272
Handler_read_next   14144510
Handler_read_prev   27428181
Handler_read_rnd    517674
Handler_read_rnd_next   477564083
Handler_rollback    65307
Handler_savepoint   0
Handler_savepoint_rollback  0
Handler_update  1374500
Handler_write   1141517
Innodb_buffer_pool_pages_data   413284
Innodb_buffer_pool_bytes_data   6771245056
Innodb_buffer_pool_pages_dirty  0
Innodb_buffer_pool_bytes_dirty  0
Innodb_buffer_pool_pages_flushed    26822
Innodb_buffer_pool_pages_free   12129538
Innodb_buffer_pool_pages_misc   1178
Innodb_buffer_pool_pages_total  12544000
Innodb_buffer_pool_read_ahead_rnd   0
Innodb_buffer_pool_read_ahead   393625
Innodb_buffer_pool_read_ahead_evicted   0
Innodb_buffer_pool_read_requests    209780846
Innodb_buffer_pool_reads    17092
Innodb_buffer_pool_wait_free    0
Innodb_buffer_pool_write_requests   1116742
Innodb_data_fsyncs  43748
Innodb_data_pending_fsyncs  0
Innodb_data_pending_reads   0
Innodb_data_pending_writes  0
Innodb_data_read    6733778944
Innodb_data_reads   410874
Innodb_data_writes  69325
Innodb_data_written 982449152
Innodb_dblwr_pages_written  26822
Innodb_dblwr_writes 1379
Innodb_have_atomic_builtins ON
Innodb_log_waits    0
Innodb_log_write_requests   168708
Innodb_log_writes   40134
Innodb_os_log_fsyncs    40988
Innodb_os_log_pending_fsyncs    0
Innodb_os_log_pending_writes    0
Innodb_os_log_written   103100928
Innodb_page_size    16384
Innodb_pages_created    2421
Innodb_pages_read   410863
Innodb_pages_written    26822
Innodb_row_lock_current_waits   0
Innodb_row_lock_time    4482
Innodb_row_lock_time_avg    498
Innodb_row_lock_time_max    4446
Innodb_row_lock_waits   9
Innodb_rows_deleted 4487
Innodb_rows_inserted    400198
Innodb_rows_read    529294128
Innodb_rows_updated 70047
Innodb_truncated_status_writes  0
Key_blocks_not_flushed  0
Key_blocks_unused   13396
Key_blocks_used 21
Key_read_requests   546691
Key_reads   11
Key_write_requests  9610
Key_writes  42
Last_query_cost 0.000000
Max_used_connections    90
Not_flushed_delayed_rows    0
Open_files  2
Open_streams    0
Open_table_definitions  150
Open_tables 64
Opened_files    4571
Opened_table_definitions    183
Opened_tables   1193
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_locker_lost  0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost    0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost    0
Prepared_stmt_count 0
Qcache_free_blocks  0
Qcache_free_memory  0
Qcache_hits 0
Qcache_inserts  0
Qcache_lowmem_prunes    0
Qcache_not_cached   0
Qcache_queries_in_cache 0
Qcache_total_blocks 0
Queries 1013649
Questions   844509
Rpl_status  AUTH_MASTER
Select_full_join    455
Select_full_range_join  0
Select_range    1597
Select_range_check  0
Select_scan 22952
Slave_heartbeat_period  0.000
Slave_open_temp_tables  0
Slave_received_heartbeats   0
Slave_retried_transactions  0
Slave_running   OFF
Slow_launch_threads 0
Slow_queries    11
Sort_merge_passes   13
Sort_range  62378
Sort_rows   303297
Sort_scan   1193
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher  
Ssl_cipher_list 
Ssl_client_connects 0
Ssl_connect_renegotiates    0
Ssl_ctx_verify_depth    0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts    0
Ssl_finished_connects   0
Ssl_session_cache_hits  0
Ssl_session_cache_misses    0
Ssl_session_cache_mode  NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size  0
Ssl_session_cache_timeouts  0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries  0
Ssl_verify_depth    0
Ssl_verify_mode 0
Ssl_version 
Table_locks_immediate   195055
Table_locks_waited  0
Tc_log_max_pages_used   0
Tc_log_page_size    0
Tc_log_page_waits   0
Threads_cached  5
Threads_connected   85
Threads_created 5238
Threads_running 1
Uptime  12333
Uptime_since_flush_status   12333

The results for show variables:

auto_increment_increment    1
auto_increment_offset   1
autocommit  ON
automatic_sp_privileges ON
back_log    50
basedir /usr/local/mysql
big_tables  OFF
binlog_cache_size   32768
binlog_direct_non_transactional_updates OFF
binlog_format   MIXED
binlog_stmt_cache_size  32768
bulk_insert_buffer_size 8388608
character_set_client    utf8mb4
character_set_connection    utf8mb4
character_set_database  latin1
character_set_filesystem    binary
character_set_results   utf8mb4
character_set_server    utf8
character_set_system    utf8
character_sets_dir  /usr/local/mysql/share/charsets/
collation_connection    utf8mb4_general_ci
collation_database  latin1_swedish_ci
collation_server    utf8_general_ci
completion_type NO_CHAIN
concurrent_insert   AUTO
connect_timeout 10
datadir /usr/local/mysql/data/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_storage_engine  InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit    100
delayed_insert_timeout  300
delayed_queue_size  1000
div_precision_increment 4
engine_condition_pushdown   ON
error_count 0
event_scheduler OFF
expire_logs_days    0
external_user   
flush   OFF
flush_time  0
foreign_key_checks  ON
ft_boolean_syntax   + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit    20
ft_stopword_file    (built-in)
general_log OFF
general_log_file    /usr/local/mysql/data/LIS3.log
group_concat_max_len    1024
have_compress   YES
have_crypt  YES
have_csv    YES
have_dynamic_loading    YES
have_geometry   YES
have_innodb YES
have_ndbcluster NO
have_openssl    DISABLED
have_partitioning   YES
have_profiling  YES
have_query_cache    YES
have_rtree_keys YES
have_ssl    DISABLED
have_symlink    YES
hostname    LIS3
identity    0
ignore_builtin_innodb   OFF
init_connect    
init_file   
init_slave  
innodb_adaptive_flushing    ON
innodb_adaptive_hash_index  ON
innodb_additional_mem_pool_size 8388608
innodb_autoextend_increment 8
innodb_autoinc_lock_mode    1
innodb_buffer_pool_instances    1
innodb_buffer_pool_size 205520896000
innodb_change_buffering all
innodb_checksums    ON
innodb_commit_concurrency   0
innodb_concurrency_tickets  500
innodb_data_file_path   ibdata1:10M:autoextend
innodb_data_home_dir    
innodb_doublewrite  ON
innodb_fast_shutdown    1
innodb_file_format  Antelope
innodb_file_format_check    ON
innodb_file_format_max  Antelope
innodb_file_per_table   OFF
innodb_flush_log_at_trx_commit  1
innodb_flush_method 
innodb_force_load_corrupted OFF
innodb_force_recovery   0
innodb_io_capacity  2000
innodb_large_prefix OFF
innodb_lock_wait_timeout    50
innodb_locks_unsafe_for_binlog  OFF
innodb_log_buffer_size  8388608
innodb_log_file_size    5242880
innodb_log_files_in_group   2
innodb_log_group_home_dir   ./
innodb_max_dirty_pages_pct  75
innodb_max_purge_lag    0
innodb_mirrored_log_groups  1
innodb_old_blocks_pct   37
innodb_old_blocks_time  0
innodb_open_files   300
innodb_print_all_deadlocks  OFF
innodb_purge_batch_size 20
innodb_purge_threads    0
innodb_random_read_ahead    OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads  64
innodb_replication_delay    0
innodb_rollback_on_timeout  OFF
innodb_rollback_segments    128
innodb_spin_wait_delay  6
innodb_stats_method nulls_equal
innodb_stats_on_metadata    OFF
innodb_stats_sample_pages   8
innodb_strict_mode  OFF
innodb_support_xa   ON
innodb_sync_spin_loops  30
innodb_table_locks  ON
innodb_thread_concurrency   0
innodb_thread_sleep_delay   10000
innodb_use_native_aio   ON
innodb_use_sys_malloc   ON
innodb_version  5.5.56
innodb_write_io_threads 64
insert_id   0
interactive_timeout 28800
join_buffer_size    131072
keep_files_on_create    OFF
key_buffer_size 16777216
key_cache_age_threshold 300
key_cache_block_size    1024
key_cache_division_limit    100
large_files_support ON
large_page_size 0
large_pages OFF
last_insert_id  0
lc_messages en_US
lc_messages_dir /usr/local/mysql/share/
lc_time_names   en_US
license GPL
local_infile    ON
lock_wait_timeout   31536000
locked_in_memory    OFF
log OFF
log_bin ON
log_bin_trust_function_creators OFF
log_error   /usr/local/mysql/data/LIS3.err
log_output  FILE
log_queries_not_using_indexes   OFF
log_slave_updates   OFF
log_slow_queries    OFF
log_warnings    1
long_query_time 10.000000
low_priority_updates    OFF
lower_case_file_system  OFF
lower_case_table_names  1
max_allowed_packet  1048576
max_binlog_cache_size   18446744073709547520
max_binlog_size 1073741824
max_binlog_stmt_cache_size  18446744073709547520
max_connect_errors  10
max_connections 800
max_delayed_threads 20
max_error_count 64
max_heap_table_size 4026531840
max_insert_delayed_threads  20
max_join_size   18446744073709551615
max_length_for_sort_data    1024
max_long_data_size  1048576
max_prepared_stmt_count 16382
max_relay_log_size  0
max_seeks_for_key   18446744073709551615
max_sort_length 1024
max_sp_recursion_depth  0
max_tmp_tables  32
max_user_connections    0
max_write_lock_count    18446744073709551615
metadata_locks_cache_size   1024
min_examined_row_limit  0
multi_range_count   256
myisam_data_pointer_size    6
myisam_max_sort_file_size   9223372036853727232
myisam_mmap_size    18446744073709551615
myisam_recover_options  OFF
myisam_repair_threads   1
myisam_sort_buffer_size 8388608
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
net_buffer_length   8192
net_read_timeout    30
net_retry_count 10
net_write_timeout   60
new OFF
old OFF
old_alter_table OFF
old_passwords   OFF
open_files_limit    4000
optimizer_prune_level   1
optimizer_search_depth  62
optimizer_switch    index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on
performance_schema  OFF
performance_schema_events_waits_history_long_size   10000
performance_schema_events_waits_history_size    10
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances   1000
performance_schema_max_file_classes 50
performance_schema_max_file_handles 32768
performance_schema_max_file_instances   10000
performance_schema_max_mutex_classes    200
performance_schema_max_mutex_instances  1000000
performance_schema_max_rwlock_classes   30
performance_schema_max_rwlock_instances 1000000
performance_schema_max_table_handles    100000
performance_schema_max_table_instances  50000
performance_schema_max_thread_classes   50
performance_schema_max_thread_instances 1000
pid_file    /usr/local/mysql/data/LIS3.pid
plugin_dir  /usr/local/mysql/lib/plugin/
port    3306
preload_buffer_size 32768
profiling   OFF
profiling_history_size  15
protocol_version    10
proxy_user  
pseudo_slave_mode   OFF
pseudo_thread_id    62
query_alloc_block_size  8192
query_cache_limit   1048576
query_cache_min_res_unit    4096
query_cache_size    0
query_cache_type    ON
query_cache_wlock_invalidate    OFF
query_prealloc_size 8192
rand_seed1  0
rand_seed2  0
range_alloc_block_size  4096
read_buffer_size    262144
read_only   OFF
read_rnd_buffer_size    524288
relay_log   
relay_log_index 
relay_log_info_file relay-log.info
relay_log_purge ON
relay_log_recovery  OFF
relay_log_space_limit   0
report_host 
report_password 
report_port 3306
report_user 
rpl_recovery_rank   0
secure_auth OFF
secure_file_priv    NULL
server_id   1
skip_external_locking   ON
skip_name_resolve   OFF
skip_networking OFF
skip_show_database  OFF
slave_compressed_protocol   OFF
slave_exec_mode STRICT
slave_load_tmpdir   /tmp
slave_max_allowed_packet    1073741824
slave_net_timeout   3600
slave_skip_errors   OFF
slave_transaction_retries   10
slave_type_conversions  
slow_launch_time    2
slow_query_log  OFF
slow_query_log_file /usr/local/mysql/data/LIS3-slow.log
socket  /tmp/mysql.sock
sort_buffer_size    524288
sql_auto_is_null    OFF
sql_big_selects ON
sql_big_tables  OFF
sql_buffer_result   OFF
sql_log_bin ON
sql_log_off OFF
sql_low_priority_updates    OFF
sql_max_join_size   18446744073709551615
sql_mode    
sql_notes   ON
sql_quote_show_create   ON
sql_safe_updates    ON
sql_select_limit    18446744073709551615
sql_slave_skip_counter  0
sql_warnings    OFF
ssl_ca  
ssl_capath  
ssl_cert    
ssl_cipher  
ssl_key 
storage_engine  InnoDB
stored_program_cache    256
sync_binlog 0
sync_frm    ON
sync_master_info    0
sync_relay_log  0
sync_relay_log_info 0
system_time_zone    UTC
table_definition_cache  400
table_open_cache    64
thread_cache_size   256
thread_concurrency  10
thread_handling one-thread-per-connection
thread_stack    262144
time_format %H:%i:%s
time_zone   SYSTEM
timed_mutexes   OFF
timestamp   1633998465
tmp_table_size  4026531840
tmpdir  /tmp
transaction_alloc_block_size    8192
transaction_prealloc_size   4096
tx_isolation    REPEATABLE-READ
unique_checks   ON
updatable_views_with_limit  YES
version 5.5.56-log
version_comment MySQL Community Server (GPL)
version_compile_machine x86_64
version_compile_os  linux-glibc2.5
wait_timeout    28800
warning_count   0

Update: The SHOW GLOBAL STATUS after 24 hours:

Aborted_clients 165641
Aborted_connects    151343
Binlog_cache_disk_use   110
Binlog_cache_use    399751
Binlog_stmt_cache_disk_use  0
Binlog_stmt_cache_use   6
Bytes_received  3164143555
Bytes_sent  29089291705
Com_admin_commands  9175854
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function  0
Com_alter_procedure 0
Com_alter_server    0
Com_alter_table 0
Com_alter_tablespace    0
Com_analyze 0
Com_assign_to_keycache  0
Com_begin   0
Com_binlog  0
Com_call_procedure  0
Com_change_db   28526
Com_change_master   0
Com_check   0
Com_checksum    0
Com_commit  51721
Com_create_db   0
Com_create_event    0
Com_create_function 0
Com_create_index    0
Com_create_procedure    0
Com_create_server   0
Com_create_table    5
Com_create_trigger  0
Com_create_udf  0
Com_create_user 1
Com_create_view 0
Com_dealloc_sql 0
Com_delete  116
Com_delete_multi    0
Com_do  0
Com_drop_db 0
Com_drop_event  0
Com_drop_function   0
Com_drop_index  0
Com_drop_procedure  0
Com_drop_server 0
Com_drop_table  5
Com_drop_trigger    0
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_execute_sql 0
Com_flush   6
Com_grant   1
Com_ha_close    0
Com_ha_open 0
Com_ha_read 0
Com_help    0
Com_insert  463710
Com_insert_select   0
Com_install_plugin  0
Com_kill    3
Com_load    0
Com_lock_tables 0
Com_optimize    0
Com_preload_keys    0
Com_prepare_sql 0
Com_purge   0
Com_purge_before_date   0
Com_release_savepoint   0
Com_rename_table    0
Com_rename_user 0
Com_repair  0
Com_replace 553
Com_replace_select  0
Com_reset   0
Com_resignal    0
Com_revoke  0
Com_revoke_all  0
Com_rollback    5018166
Com_rollback_to_savepoint   0
Com_savepoint   0
Com_select  3530506
Com_set_option  6108267
Com_show_authors    0
Com_show_binlog_events  0
Com_show_binlogs    0
Com_show_charsets   83
Com_show_collations 202
Com_show_contributors   0
Com_show_create_db  0
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_proc    0
Com_show_create_table   2622876
Com_show_create_trigger 0
Com_show_databases  32
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  0
Com_show_errors 0
Com_show_events 0
Com_show_fields 623
Com_show_function_status    25
Com_show_grants 1
Com_show_keys   175
Com_show_master_status  0
Com_show_open_tables    0
Com_show_plugins    0
Com_show_privileges 0
Com_show_procedure_status   25
Com_show_processlist    7
Com_show_profile    0
Com_show_profiles   0
Com_show_relaylog_events    0
Com_show_slave_hosts    0
Com_show_slave_status   0
Com_show_status 108
Com_show_storage_engines    20
Com_show_table_status   0
Com_show_tables 48
Com_show_triggers   0
Com_show_variables  154637
Com_show_warnings   0
Com_signal  0
Com_slave_start 0
Com_slave_stop  0
Com_stmt_close  61
Com_stmt_execute    61
Com_stmt_fetch  0
Com_stmt_prepare    61
Com_stmt_reprepare  0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_truncate    4
Com_uninstall_plugin    0
Com_unlock_tables   0
Com_update  101555
Com_update_multi    0
Com_xa_commit   0
Com_xa_end  0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression OFF
Connections 1508463
Created_tmp_disk_tables 3129
Created_tmp_files   212
Created_tmp_tables  205318
Delayed_errors  0
Delayed_insert_threads  0
Delayed_writes  0
Flush_commands  1
Handler_commit  2917483
Handler_delete  25545
Handler_discover    0
Handler_prepare 1158436
Handler_read_first  303544
Handler_read_key    119357180
Handler_read_last   1448
Handler_read_next   56965743323
Handler_read_prev   146042061
Handler_read_rnd    1947683
Handler_read_rnd_next   29332316484
Handler_rollback    1155199
Handler_savepoint   0
Handler_savepoint_rollback  0
Handler_update  8766598
Handler_write   7548336
Innodb_buffer_pool_bytes_data   9527721984
Innodb_buffer_pool_bytes_dirty  0
Innodb_buffer_pool_pages_data   581526
Innodb_buffer_pool_pages_dirty  0
Innodb_buffer_pool_pages_flushed    449839
Innodb_buffer_pool_pages_free   11957068
Innodb_buffer_pool_pages_misc   5405
Innodb_buffer_pool_pages_total  12543999
Innodb_buffer_pool_read_ahead   521479
Innodb_buffer_pool_read_ahead_evicted   0
Innodb_buffer_pool_read_ahead_rnd   0
Innodb_buffer_pool_read_requests    12573857453
Innodb_buffer_pool_reads    49143
Innodb_buffer_pool_wait_free    0
Innodb_buffer_pool_write_requests   4032597
Innodb_data_fsyncs  813966
Innodb_data_pending_fsyncs  0
Innodb_data_pending_reads   0
Innodb_data_pending_writes  0
Innodb_data_read    9389953024
Innodb_data_reads   572994
Innodb_data_writes  1254814
Innodb_data_written 15454642688
Innodb_dblwr_pages_written  449839
Innodb_dblwr_writes 12116
Innodb_have_atomic_builtins ON
Innodb_log_waits    0
Innodb_log_write_requests   658731
Innodb_log_writes   780456
Innodb_os_log_fsyncs    789652
Innodb_os_log_pending_fsyncs    0
Innodb_os_log_pending_writes    0
Innodb_os_log_written   709616128
Innodb_page_size    16384
Innodb_pages_created    8543
Innodb_pages_read   572983
Innodb_pages_written    449839
Innodb_row_lock_current_waits   0
Innodb_row_lock_time    196248
Innodb_row_lock_time_avg    2393
Innodb_row_lock_time_max    50420
Innodb_row_lock_waits   82
Innodb_rows_deleted 25544
Innodb_rows_inserted    486543
Innodb_rows_read    82917832779
Innodb_rows_updated 226041
Innodb_truncated_status_writes  0
Key_blocks_not_flushed  0
Key_blocks_unused   13396
Key_blocks_used 33
Key_read_requests   1558852
Key_reads   55
Key_write_requests  43122
Key_writes  6
Last_query_cost 0.000000
Max_used_connections    156
Not_flushed_delayed_rows    0
Open_files  3
Open_streams    0
Open_table_definitions  240
Open_tables 64
Opened_files    14475
Opened_table_definitions    249
Opened_tables   9158
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_locker_lost  0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost    0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost    0
Prepared_stmt_count 0
Qcache_free_blocks  0
Qcache_free_memory  0
Qcache_hits 0
Qcache_inserts  0
Qcache_lowmem_prunes    0
Qcache_not_cached   0
Qcache_queries_in_cache 0
Qcache_total_blocks 0
Queries 28640951
Questions   19281984
Rpl_status  AUTH_MASTER
Select_full_join    811
Select_full_range_join  0
Select_range    32153
Select_range_check  0
Select_scan 475471
Slave_heartbeat_period  0.000
Slave_open_temp_tables  0
Slave_received_heartbeats   0
Slave_retried_transactions  0
Slave_running   OFF
Slow_launch_threads 0
Slow_queries    46
Sort_merge_passes   53
Sort_range  1099066
Sort_rows   2462510
Sort_scan   6214
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher  
Ssl_cipher_list 
Ssl_client_connects 0
Ssl_connect_renegotiates    0
Ssl_ctx_verify_depth    0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts    0
Ssl_finished_connects   0
Ssl_session_cache_hits  0
Ssl_session_cache_misses    0
Ssl_session_cache_mode  NONE
Ssl_session_cache_overflows 0
Ssl_session_cache_size  0
Ssl_session_cache_timeouts  0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries  0
Ssl_verify_depth    0
Ssl_verify_mode 0
Ssl_version 
Table_locks_immediate   2298466
Table_locks_waited  0
Tc_log_max_pages_used   0
Tc_log_page_size    0
Tc_log_page_waits   0
Threads_cached  0
Threads_connected   62
Threads_created 1508462
Threads_running 3
Uptime  96644
Uptime_since_flush_status   96644

Answer 1

Analysis of GLOBAL STATUS and VARIABLES:

Observations:

• Version: 5.5.56-log
• 378 GB of RAM
• Uptime = 1d 02:50:44
• 296 Queries/sec : 200 Questions/sec

The More Important Issues:

table_open_cache = 5000
innodb_buffer_pool_instances = 16
innodb_log_file_size = 20M   -- Caution; this will be hard to change in 5.5, so skip it until you upgrade
read_buffer_size = 1M
query_cache_type = OFF
query_cache_size = 0

Version 5.5 is old; 5.6 has table_open_cache_hits and misses, which will help tune table_open_cache.

The machine seems to have far more RAM than you need. (32GB would probably be sufficient.)

There were a lot of SHOWs, SETs, ROLLBACKs, and admin commands; any idea what is going on?

Recommend setting up the slowlog as a step toward investigating the long row_log times and other issues. http://mysql.rjweb.org/doc.php/mysql_analysis#slow_queries_and_slowlog

Details and other observations:

( table_open_cache ) = 64 -- Number of table descriptors to cache -- Several hundred is usually good.

( Com_show_create_table ) = 2,622,876 / 96644 = 27 /sec -- How often SHOW CREATE TABLE is executed. Possibly comes from a 3rd party package that is sloppy. -- Check table_open_cache (now 64) and open_files_limit (now 4000). Or avoid the 3rd party package.

( innodb_buffer_pool_size ) = 196,000 / 387072M = 50.6% -- % of RAM used for InnoDB buffer_pool -- Set to about 70% of available RAM. (To low is less efficient; too high risks swapping.)

( innodb_buffer_pool_size / innodb_buffer_pool_instances ) = 196000M / 1 = 196000MB -- Size of each buffer_pool instance. -- An instance should be at least 1GB. In very large RAM, have 16 instances.

( Innodb_buffer_pool_pages_free * 16384 / innodb_buffer_pool_size ) = 11,957,068 * 16384 / 196000M = 95.3% -- buffer pool free -- buffer_pool_size is bigger than working set; could decrease it

( Innodb_buffer_pool_pages_free / Innodb_buffer_pool_pages_total ) = 11,957,068 / 12543999 = 95.3% -- Pct of buffer_pool currently not in use -- innodb_buffer_pool_size (now 205520896000) is bigger than necessary?

( Innodb_buffer_pool_bytes_data / innodb_buffer_pool_size ) = 9,527,721,984 / 196000M = 4.6% -- Percent of buffer pool taken up by data -- A small percent may indicate that the buffer_pool is unnecessarily big.

( innodb_log_buffer_size / innodb_log_file_size ) = 8M / 5M = 160.0% -- Buffer is in RAM; file is on disk. -- The buffer_size should be smaller and/or the file_size should be larger.

( Innodb_os_log_written / (Uptime / 3600) / innodb_log_files_in_group / innodb_log_file_size ) = 709,616,128 / (96644 / 3600) / 2 / 5M = 2.52 -- Ratio -- (see minutes)

( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 96,644 / 60 * 5M / 709616128 = 11.9 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf. -- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size (now 5242880). (Cannot change in AWS.)

( innodb_flush_method ) = innodb_flush_method = -- How InnoDB should ask the OS to write blocks. Suggest O_DIRECT or O_ALL_DIRECT (Percona) to avoid double buffering. (At least for Unix.) See chrischandler for caveat about O_ALL_DIRECT

( Com_rollback ) = 5,018,166 / 96644 = 52 /sec -- ROLLBACKs in InnoDB. -- An excessive frequency of rollbacks may indicate inefficient app logic.

( Handler_rollback ) = 1,155,199 / 96644 = 12 /sec -- Why so many rollbacks?

( innodb_file_per_table ) = innodb_file_per_table = OFF -- Put each file in its own tablespace -- (Mildly recommended, especially for large tables)

( Innodb_row_lock_time_avg ) = 2,393 -- Avg time to lock a row (millisec) -- Possibly conflicting queries; possibly table scans.

( Innodb_row_lock_time_max ) = 50,420 -- Max time to lock a row (millisec) -- Possibly conflicting queries; possibly table scans.

( innodb_strict_mode ) = innodb_strict_mode = OFF -- Catches some subtle errors earlier. -- OFF leaves some warnings as warnings; ON makes them errors.

( sync_binlog ) = 0 -- Use 1 for added security, at some cost of I/O =1 may lead to lots of "query end"; =0 may lead to "binlog at impossible position" and lose transactions in a crash, but is faster. 0 is OK for Galera.

( innodb_adaptive_hash_index ) = innodb_adaptive_hash_index = ON -- Usually should be ON. -- There are cases where OFF is better. See also innodb_adaptive_hash_index_parts (after 5.7.9) and innodb_adaptive_hash_index_partitions (MariaDB and Percona). ON has been implicated in rare crashes (bug 73890). 10.5.0 decided to default OFF.

( innodb_print_all_deadlocks ) = innodb_print_all_deadlocks = OFF -- Whether to log all Deadlocks. -- If you are plagued with Deadlocks, turn this on. Caution: If you have lots of deadlocks, this may write a lot to disk.

( character_set_server ) = character_set_server = utf8 -- Charset problems may be helped by setting character_set_server (now utf8) to utf8mb4. That is the future default.

( (Com_show_create_table + Com_show_fields) / Questions ) = (2622876 + 623) / 19281984 = 13.6% -- Naughty framework -- spending a lot of effort rediscovering the schema. -- Complain to the 3rd party vendor.

( local_infile ) = local_infile = ON -- local_infile (now ON) = ON is a potential security issue

( Handler_read_rnd_next / Com_select ) = 29,332,316,484 / 3530506 = 8,308 -- Avg rows scanned per SELECT. (approx) -- Consider raising read_buffer_size (now 262144)

( Com_rollback / (Com_commit + Com_rollback) ) = 5,018,166 / (51721 + 5018166) = 99.0% -- Rollback : Commit ratio -- Rollbacks are costly; change app logic

( Com_show_variables ) = 154,637 / 96644 = 1.6 /sec -- SHOW VARIABLES ... -- Why are you requesting the VARIABLES so often?

( Com_admin_commands ) = 9,175,854 / 96644 = 95 /sec -- Why so many DDL statements?

( Com_admin_commands / Queries ) = 9,175,854 / 28640951 = 32.0% -- Percent of queries that are "admin" commands. -- What's going on?

( Com_set_option / Com_select ) = 6,108,267 / 3530506 = 173.0% -- It seems 'wrong' to do more SETs than SELECTs.

( Com__biggest ) = Com__biggest = Com_admin_commands -- Which of the "Com_" metrics is biggest. -- Normally it is Com_select (now 3530506). If something else, then it may be a sloppy platform, or may be something else.

( Com_show__star / Questions ) = 2,778,862 / 19281984 = 14.4% -- Pct of queries that are SHOW commands. -- Why so many SHOW commands?

( binlog_format ) = binlog_format = MIXED -- STATEMENT/ROW/MIXED. -- ROW is preferred by 5.7 (10.3)

( innodb_autoinc_lock_mode ) = 1 -- Galera: desires 2 -- 2 = "interleaved"; 1 = "consecutive" is typical; 0 = "traditional". -- Galera desires 2; 2 requires BINLOG_FORMAT=ROW or MIXED

( log_slow_queries ) = log_slow_queries = OFF -- Whether to log slow queries. (Before 5.1.29, 5.6.1)

( slow_query_log ) = slow_query_log = OFF -- Whether to log slow queries. (5.1.12)

( long_query_time ) = 10 -- Cutoff (Seconds) for defining a "slow" query. -- Suggest 2

( back_log ) = 50 -- (Autosized as of 5.6.6; based on max_connections) -- Raising to min(150, max_connections (now 800)) may help when doing lots of connections.

( Max_used_connections / max_connections ) = 156 / 800 = 19.5% -- Peak % of connections -- Since several memory factors can expand based on max_connections (now 800), it is good not to have that setting too high.

( Aborted_clients ) = 165,641 / 96644 = 1.7 /sec -- Threads bumped due to wait_timeout -- Increase wait_timeout (now 28800); be nice, use disconnect

( Aborted_connects / Connections ) = 151,343 / 1508463 = 10.0% -- Perhaps a hacker is trying to break in? (Attempts to connect)

( Aborted_connects ) = 151,343 / 96644 = 1.6 /sec -- Could not get a connection, or hacker ? (Attempts to connect) -- Raise max_connections (now 800)

( Connections ) = 1,508,463 / 96644 = 16 /sec -- Connections -- Increase wait_timeout (now 28800); use pooling?

( thread_cache_size / Max_used_connections ) = 256 / 156 = 164.1% -- There is no advantage in having the thread cache bigger than your likely number of connections. Wasting space is the disadvantage.

You have the Query Cache half-off. You should set both query_cache_type = OFF and query_cache_size = 0 . There is (according to a rumor) a 'bug' in the QC code that leaves some code on unless you turn off both of those settings.

Abnormally small:

Open_tables = 64
net_buffer_length = 8,192
table_open_cache / max_connections = 0.08

Abnormally large:

Com_create_user = 0.037 /HR
Com_show_charsets = 3.1 /HR
Com_show_storage_engines = 0.75 /HR
Handler_read_next = 589439 /sec
Handler_read_next / Handler_read_key = 477
Handler_read_rnd_next / Handler_read_rnd = 15,060
Innodb_buffer_pool_pages_free = 1.2e+7
Innodb_buffer_pool_pages_total = 1.25e+7
Innodb_rows_read = 857971 /sec
innodb_read_io_threads = 64
innodb_write_io_threads = 64
max_heap_table_size = 3840MB
min(max_heap_table_size, tmp_table_size) = 3840MB

Abnormal strings:

innodb_fast_shutdown = 1
lower_case_table_names = 1
sql_safe_updates = ON

Answer 2

5.5 is 3 major upgrades behind. There will be hiccups. You must do 3 separate upgrades: 5.5 -> 5.6 -> 5.7 -> 8.0. The last step will be the most painful.

Can your colleagues identify the queries that go slower now? There could be some clues there.

A minor improvement: innodb_buffer_pool_instances = 16

Please refresh the GLOBAL STATUS after it has been running at least 24 hours.