'inconsistent during the day' Mave Nov 14 at 20:30
my.cnf things to do:
in [mysqld] section,
max_heap_table_size = 24M # from 16M
tmp_table_size = 24M # from 16M to reduce 38% HDD used today
innodb_io_capacity = 2000 # from 200 for additional capacity
innodb_io_capacity = 20000 # from 2000 for additional capacity
read_rnd_buffer_size = 128K # from 256K to reduce 4738 RPS
delay_key_write = OFF # from ON to speed storage and minimize corruption exposure
key_buffer_size = 100K # from 500+ M, used only by MyISAM
innodb_change_buffer_max_size = 10 # from 25%, less than 3% max used in 20 days
innodb_read_ahead_threshold = 8 # from 56 rather than waiting
innodb_stats_sample_pages = 32 # from 8 to improve cardinality data for optimizer
max_seeks_for_key = 64 # from a huge number to limit wasted dives
max_write_lock_count = 5 # from a huge number to give RD a chance quicker
query_cache_limit = 1M # from ~ 104K for EACH QC candidate size limit
query_cache_min_res_unit = 512 # from 4096 to pack more QC results into QC
query_prealloc_size = 32K # from 8K to avoid acquiring RAM all day for parsing
Typical HARD rules, only one a day and be sure before moving to next attempt to smooth your performance.
Problems to be addressed related to the 20 days of status information, A. why 446,000 com_rollback_to savepoints ? B. why 2,444 com_rollback ? C. normally Com_begin has a companion Com_commit. 4,283 short on commit. D. 196 com_flush side effect is destruction of much of Query Cache results E. Failure to have innodb_print_all_deadlocks = ON means you will never know what caused the problem or figure it out. F. ~ 5 million select_scan usually means an index is missing that could speed process G. only 1 of each 7 candidate query results made it into Query Cache. One minute of your general log will enable analysis to determine if this ratio could be improved significantly.
Hoping you are making progress on raising ulimit capacity within Linux. This will allow more 'hits' than current ulimit's.
So much of your effort has been EXCELLENT and these suggestions in no way are critical of where you are today. Need your feedback, when time permits. Not every DAY.