0

We've got a MySQL database (on Amazon RDS) which occasionally gets a lot of INSERT traffic with large (~1MB) TEXT columns. We find that when this happens, this causes the database CPU to get quite high, with a "see-saw" pattern. This causes other queries that would ordinarily be very fast to slow down unacceptably.

I synthesised a benchmark where I did nothing except INSERT large TEXT columns, with no other work happening. The CPU usage profile ends up looking like this:

see-saw CPU pattern

My best guess is that there is some sort of page allocation algorithm strategy that kicks in every couple of minutes as data is being added and runs in some critical single-threaded part of MySQL. Can anybody suggest how to tune this better?

More details:

The table we are inserting into:

mysql> show create table workItemContentVersion \G
*************************** 1. row ***************************
       Table: workItemContentVersion
Create Table: CREATE TABLE `workItemContentVersion` (
  `parentId` int(11) NOT NULL,
  `serialNumber` int(11) NOT NULL,
  `previousVersionSerialNumber` int(11) DEFAULT NULL,
  `authorPrincipalName` varchar(256) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `textAnswer` longtext COLLATE utf8mb4_unicode_ci,
  `textAnswerContentType` varchar(10) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
  `blobAnswer` longblob,
  `contentSavedOn` datetime DEFAULT NULL,
  PRIMARY KEY (`parentId`,`serialNumber`),
  CONSTRAINT `fk_workItemContentVersion_workItem` FOREIGN KEY (`parentId`) REFERENCES `workItem` (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci
1 row in set (0.00 sec)

The INSERT statement being used for the benchmark (where the parameter for textAnswer is 1MB of the letter 'a':

INSERT IGNORE INTO workItemContentVersion (parentId, serialNumber, textAnswer)
VALUES (?, ?, ?), (?, ?, ?)

Hardware details: Amazon RDS on db.r3.xlarge instance (4 vCPU, 30GB RAM, 500 Mbps disk bandwidth)

Server config (output of SHOW GLOBAL STATUS):

Variable_name   Value
Aborted_clients 6
Aborted_connects    0
Binlog_cache_disk_use   268012
Binlog_cache_use    268284
Binlog_stmt_cache_disk_use  0
Binlog_stmt_cache_use   0
Bytes_received  514070955062
Bytes_sent  41617844
Com_admin_commands  0
Com_assign_to_keycache  0
Com_alter_db    0
Com_alter_db_upgrade    0
Com_alter_event 0
Com_alter_function  0
Com_alter_procedure 0
Com_alter_server    0
Com_alter_table 0
Com_alter_tablespace    0
Com_alter_user  0
Com_analyze 0
Com_begin   0
Com_binlog  0
Com_call_procedure  0
Com_change_db   21994
Com_change_master   0
Com_check   0
Com_checksum    0
Com_commit  272
Com_create_db   0
Com_create_event    0
Com_create_function 0
Com_create_index    0
Com_create_procedure    0
Com_create_server   0
Com_create_table    0
Com_create_trigger  0
Com_create_udf  0
Com_create_user 0
Com_create_view 0
Com_dealloc_sql 0
Com_delete  0
Com_delete_multi    0
Com_do  0
Com_drop_db 0
Com_drop_event  0
Com_drop_function   0
Com_drop_index  0
Com_drop_procedure  0
Com_drop_server 0
Com_drop_table  0
Com_drop_trigger    0
Com_drop_user   0
Com_drop_view   0
Com_empty_query 0
Com_execute_sql 0
Com_flush   278
Com_get_diagnostics 0
Com_grant   0
Com_ha_close    0
Com_ha_open 0
Com_ha_read 0
Com_help    0
Com_insert  268288
Com_insert_select   0
Com_install_plugin  0
Com_kill    1
Com_load    0
Com_lock_tables 0
Com_optimize    0
Com_preload_keys    0
Com_prepare_sql 0
Com_purge   278
Com_purge_before_date   0
Com_release_savepoint   0
Com_rename_table    0
Com_rename_user 0
Com_repair  0
Com_replace 0
Com_replace_select  0
Com_reset   0
Com_resignal    0
Com_revoke  0
Com_revoke_all  0
Com_rollback    0
Com_rollback_to_savepoint   0
Com_savepoint   0
Com_select  132852
Com_set_option  8
Com_signal  0
Com_show_binlog_events  0
Com_show_binlogs    0
Com_show_charsets   0
Com_show_collations 0
Com_show_create_db  0
Com_show_create_event   0
Com_show_create_func    0
Com_show_create_proc    0
Com_show_create_table   2
Com_show_create_trigger 0
Com_show_databases  2
Com_show_engine_logs    0
Com_show_engine_mutex   0
Com_show_engine_status  0
Com_show_events 0
Com_show_errors 0
Com_show_fields 81
Com_show_function_code  0
Com_show_function_status    0
Com_show_grants 0
Com_show_keys   0
Com_show_master_status  0
Com_show_open_tables    0
Com_show_plugins    0
Com_show_privileges 0
Com_show_procedure_code 0
Com_show_procedure_status   0
Com_show_processlist    5
Com_show_profile    0
Com_show_profiles   0
Com_show_relaylog_events    0
Com_show_slave_hosts    0
Com_show_slave_status   0
Com_show_status 267
Com_show_storage_engines    0
Com_show_table_status   0
Com_show_tables 2
Com_show_triggers   0
Com_show_variables  1
Com_show_warnings   0
Com_slave_start 0
Com_slave_stop  0
Com_stmt_close  15
Com_stmt_execute    270641
Com_stmt_fetch  0
Com_stmt_prepare    50
Com_stmt_reprepare  0
Com_stmt_reset  0
Com_stmt_send_long_data 0
Com_truncate    0
Com_uninstall_plugin    0
Com_unlock_tables   0
Com_update  0
Com_update_multi    0
Com_xa_commit   0
Com_xa_end  0
Com_xa_prepare  0
Com_xa_recover  0
Com_xa_rollback 0
Com_xa_start    0
Compression OFF
Connection_errors_accept    0
Connection_errors_internal  0
Connection_errors_max_connections   0
Connection_errors_peer_address  0
Connection_errors_select    0
Connection_errors_tcpwrap   0
Connections 22288
Created_tmp_disk_tables 1391
Created_tmp_files   22009
Created_tmp_tables  55230
Delayed_errors  0
Delayed_insert_threads  0
Delayed_writes  0
Flush_commands  1
Handler_commit  545226
Handler_delete  0
Handler_discover    0
Handler_external_lock   554039
Handler_mrr_init    0
Handler_prepare 536574
Handler_read_first  8623
Handler_read_key    8936
Handler_read_last   20
Handler_read_next   1
Handler_read_prev   594
Handler_read_rnd    0
Handler_read_rnd_next   125997
Handler_rollback    0
Handler_savepoint   0
Handler_savepoint_rollback  0
Handler_update  272
Handler_write   567816
Innodb_buffer_pool_dump_status  not started
Innodb_buffer_pool_load_status  not started
Innodb_buffer_pool_pages_data   1455843
Innodb_buffer_pool_bytes_data   23852531712
Innodb_buffer_pool_pages_dirty  11311
Innodb_buffer_pool_bytes_dirty  185319424
Innodb_buffer_pool_pages_flushed    32307952
Innodb_buffer_pool_pages_free   7324
Innodb_buffer_pool_pages_misc   1
Innodb_buffer_pool_pages_total  1463168
Innodb_buffer_pool_read_ahead_rnd   0
Innodb_buffer_pool_read_ahead   0Warning: Using a password on the command line interface can be insecure.

Innodb_buffer_pool_read_ahead_evicted   0
Innodb_buffer_pool_read_requests    231352753
Innodb_buffer_pool_reads    5484
Innodb_buffer_pool_wait_free    0
Innodb_buffer_pool_write_requests   226317402
Innodb_data_fsyncs  1881794
Innodb_data_pending_fsyncs  2
Innodb_data_pending_reads   0
Innodb_data_pending_writes  1
Innodb_data_read    92016640
Innodb_data_reads   5806
Innodb_data_writes  33848583
Innodb_data_written 1593306025984
Innodb_dblwr_pages_written  32308192
Innodb_dblwr_writes 645081
Innodb_have_atomic_builtins ON
Innodb_log_waits    413
Innodb_log_write_requests   1041108828
Innodb_log_writes   335941
Innodb_os_log_fsyncs    346038
Innodb_os_log_pending_fsyncs    2
Innodb_os_log_pending_writes    0
Innodb_os_log_written   534630468096
Innodb_page_size    16384
Innodb_pages_created    31897395
Innodb_pages_read   5483
Innodb_pages_written    32308192
Innodb_row_lock_current_waits   0
Innodb_row_lock_time    1874
Innodb_row_lock_time_avg    93
Innodb_row_lock_time_max    425
Innodb_row_lock_waits   20
Innodb_rows_deleted 0
Innodb_rows_inserted    513977
Innodb_rows_read    9521
Innodb_rows_updated 272
Innodb_num_open_files   42
Innodb_truncated_status_writes  0
Innodb_available_undo_logs  128
Key_blocks_not_flushed  0
Key_blocks_unused   13384
Key_blocks_used 12
Key_read_requests   12
Key_reads   12
Key_write_requests  0
Key_writes  0
Last_query_cost 0.000000
Last_query_partial_plans    0
Max_used_connections    12
Not_flushed_delayed_rows    0
Open_files  31
Open_streams    0
Open_table_definitions  114
Open_tables 176
Opened_files    54928
Opened_table_definitions    114
Opened_tables   6014
Performance_schema_accounts_lost    0
Performance_schema_cond_classes_lost    0
Performance_schema_cond_instances_lost  0
Performance_schema_digest_lost  0
Performance_schema_file_classes_lost    0
Performance_schema_file_handles_lost    0
Performance_schema_file_instances_lost  0
Performance_schema_hosts_lost   0
Performance_schema_locker_lost  0
Performance_schema_mutex_classes_lost   0
Performance_schema_mutex_instances_lost 0
Performance_schema_rwlock_classes_lost  0
Performance_schema_rwlock_instances_lost    0
Performance_schema_session_connect_attrs_lost   0
Performance_schema_socket_classes_lost  0
Performance_schema_socket_instances_lost    0
Performance_schema_stage_classes_lost   0
Performance_schema_statement_classes_lost   0
Performance_schema_table_handles_lost   0
Performance_schema_table_instances_lost 0
Performance_schema_thread_classes_lost  0
Performance_schema_thread_instances_lost    0
Performance_schema_users_lost   0
Prepared_stmt_count 10
Qcache_free_blocks  1
Qcache_free_memory  1031352
Qcache_hits 0
Qcache_inserts  0
Qcache_lowmem_prunes    0
Qcache_not_cached   132851
Qcache_queries_in_cache 0
Qcache_total_blocks 1
Queries 446669
Questions   446338
Select_full_join    0
Select_full_range_join  0
Select_range    0
Select_range_check  0
Select_scan 61056
Slave_heartbeat_period  0.000
Slave_last_heartbeat    
Slave_open_temp_tables  0
Slave_received_heartbeats   0
Slave_retried_transactions  0
Slave_running   OFF
Slow_launch_threads 0
Slow_queries    0
Sort_merge_passes   0
Sort_range  0
Sort_rows   0
Sort_scan   2780
Ssl_accept_renegotiates 0
Ssl_accepts 0
Ssl_callback_cache_hits 0
Ssl_cipher  
Ssl_cipher_list 
Ssl_client_connects 0
Ssl_connect_renegotiates    0
Ssl_ctx_verify_depth    0
Ssl_ctx_verify_mode 0
Ssl_default_timeout 0
Ssl_finished_accepts    0
Ssl_finished_connects   0
Ssl_server_not_after    
Ssl_server_not_before   
Ssl_session_cache_hits  0
Ssl_session_cache_misses    0
Ssl_session_cache_mode  Unknown
Ssl_session_cache_overflows 0
Ssl_session_cache_size  0
Ssl_session_cache_timeouts  0
Ssl_sessions_reused 0
Ssl_used_session_cache_entries  0
Ssl_verify_depth    0
Ssl_verify_mode 0
Ssl_version 
Table_locks_immediate   277022
Table_locks_waited  0
Table_open_cache_hits   323546
Table_open_cache_misses 183
Table_open_cache_overflows  0
Tc_log_max_pages_used   0
Tc_log_page_size    0
Tc_log_page_waits   0
Threads_cached  0
Threads_connected   12
Threads_created 12
Threads_running 6
Uptime  83387
Uptime_since_flush_status   83387

Output of SHOW VARIABLES:

Variable_name   Value
auto_increment_increment    1
auto_increment_offset   1
autocommit  ON
automatic_sp_privileges ON
back_log    558
basedir /rdsdbbin/mysql/
big_tables  OFF
bind_address    *
binlog_cache_size   32768
binlog_checksum CRC32
binlog_direct_non_transactional_updates OFF
binlog_format   MIXED
binlog_max_flush_queue_time 0
binlog_order_commits    ON
binlog_row_image    FULL
binlog_rows_query_log_events    OFF
binlog_stmt_cache_size  32768
binlogging_impossible_mode  IGNORE_ERROR
block_encryption_mode   aes-128-ecb
bulk_insert_buffer_size 8388608
character_set_client    latin1
character_set_connection    latin1
character_set_database  latin1
character_set_filesystem    binary
character_set_results   latin1
character_set_server    latin1
character_set_system    utf8
character_sets_dir  /rdsdbbin/mysql-5.6.21.R1/share/charsets/
collation_connection    latin1_swedish_ci
collation_database  latin1_swedish_ci
collation_server    latin1_swedish_ci
completion_type NO_CHAIN
concurrent_insert   AUTO
connect_timeout 10
core_file   OFF
datadir /rdsdbdata/db/
date_format %Y-%m-%d
datetime_format %Y-%m-%d %H:%i:%s
default_storage_engine  InnoDB
default_tmp_storage_engine  InnoDB
default_week_format 0
delay_key_write ON
delayed_insert_limit    100
delayed_insert_timeout  300
delayed_queue_size  1000
disconnect_on_expired_password  ON
div_precision_increment 4
end_markers_in_json OFF
enforce_gtid_consistency    OFF
eq_range_index_dive_limit   10
error_count 0
event_scheduler OFF
expire_logs_days    0
explicit_defaults_for_timestamp ON
external_user   
flush   OFF
flush_time  0
foreign_key_checks  ON
ft_boolean_syntax   + -><()~*:""&|
ft_max_word_len 84
ft_min_word_len 4
ft_query_expansion_limit    20
ft_stopword_file    (built-in)
general_log OFF
general_log_file    /rdsdbdata/log/general/mysql-general.log
group_concat_max_len    1024
gtid_executed   
gtid_mode   OFF
gtid_next   AUTOMATIC
gtid_owned  
gtid_purged 
have_compress   YES
have_crypt  YES
have_dynamic_loading    YES
have_geometry   YES
have_openssl    YES
have_profiling  YES
have_query_cache    YES
have_rtree_keys YES
have_ssl    YES
have_symlink    YES
host_cache_size 730
hostname    ip-172-22-1-13
identity    0
ignore_builtin_innodb   OFF
ignore_db_dirs  
init_connect    
init_file   
init_slave  
innodb_adaptive_flushing    ON
innodb_adaptive_flushing_lwm    10
innodb_adaptive_hash_index  ON
innodb_adaptive_max_sleep_delay 150000
innodb_additional_mem_pool_size 8388608
innodb_api_bk_commit_interval   5
innodb_api_disable_rowlock  OFF
innodb_api_enable_binlog    OFF
innodb_api_enable_mdl   OFF
innodb_api_trx_level    0
innodb_autoextend_increment 64
innodb_autoinc_lock_mode    1
innodb_buffer_pool_dump_at_shutdown OFF
innodb_buffer_pool_dump_now OFF
innodb_buffer_pool_filename ib_buffer_pool
innodb_buffer_pool_instances    8
innodb_buffer_pool_load_abort   OFF
innodb_buffer_pool_load_at_startup  OFF
innodb_buffer_pool_load_now OFF
innodb_buffer_pool_size 23972544512
innodb_change_buffer_max_size   25
innodb_change_buffering all
innodb_checksum_algorithm   innodb
innodb_checksums    ON
innodb_cmp_per_index_enabled    OFF
innodb_commit_concurrency   0
innodb_compression_failure_threshold_pct    5
innodb_compression_level    6
innodb_compression_pad_pct_max  50
innodb_concurrency_tickets  5000
innodb_data_file_path   ibdata1:12M:autoextend
innodb_data_home_dir    /rdsdbdata/db/innodb
innodb_disable_sort_file_cache  OFF
innodb_doublewrite  ON
innodb_fast_shutdown    1
innodb_file_format  Antelope
innodb_file_format_check    ON
innodb_file_format_max  Antelope
innodb_file_per_table   ON
innodb_flush_log_at_timeout 1
innodb_flush_log_at_trx_commit  1
innodb_flush_method 
innodb_flush_neighbors  1
innodb_flushing_avg_loops   30
innodb_force_load_corrupted OFF
innodb_force_recovery   0
innodb_ft_aux_table 
innodb_ft_cache_size    8000000
innodb_ft_enable_diag_print OFF
innodb_ft_enable_stopword   ON
innodb_ft_max_token_size    84
innodb_ft_min_token_size    3
innodb_ft_num_word_optimize 2000
innodb_ft_result_cache_limit    2000000000
innodb_ft_server_stopword_table 
innodb_ft_sort_pll_degree   2
innodb_ft_total_cache_size  640000000
innodb_ft_user_stopword_table   
innodb_io_capacity  200
innodb_io_capacity_max  2000
innodb_large_prefix OFF
innodb_lock_wait_timeout    50
innodb_locks_unsafe_for_binlog  OFF
innodb_log_buffer_size  8388608
innodb_log_compressed_pages ON
innodb_log_file_size    134217728
innodb_log_files_in_group   2
innodb_log_group_home_dir   /rdsdbdata/log/innodb
innodb_lru_scan_depth   1024
innodb_max_dirty_pages_pct  75
innodb_max_dirty_pages_pct_lwm  0
innodb_max_purge_lag    0
innodb_max_purge_lag_delay  0
innodb_mirrored_log_groups  1
innodb_monitor_disable  
innodb_monitor_enable   
innodb_monitor_reset    
innodb_monitor_reset_all    
innodb_old_blocks_pct   37
innodb_old_blocks_time  1000
innodb_online_alter_log_max_size    134217728
innodb_open_files   2000
innodb_optimize_fulltext_only   OFF
innodb_page_size    16384
innodb_print_all_deadlocks  OFF
innodb_purge_batch_size 300
innodb_purge_threads    1
innodb_random_read_ahead    OFF
innodb_read_ahead_threshold 56
innodb_read_io_threads  4
innodb_read_only    OFF
innodb_replication_delay    0
innodb_rollback_on_timeout  OFF
innodb_rollback_segments    128
innodb_sort_buffer_size 1048576
innodb_spin_wait_delay  6
innodb_stats_auto_recalc    ON
innodb_stats_method nulls_equal
innodb_stats_on_metadata    OFF
innodb_stats_persistent ON
innodb_stats_persistent_sample_pages    20
innodb_stats_sample_pages   8
innodb_stats_transient_sample_pages 8
innodb_status_output    OFF
innodb_status_output_locks  OFF
innodb_strict_mode  OFF
innodb_support_xa   ON
innodb_sync_array_size  1
innodb_sync_spin_loops  30
innodb_table_locks  ON
innodb_thread_concurrency   0
innodb_thread_sleep_delay   10000
innodb_undo_directory   .
innodb_undo_logs    128
innodb_undo_tablespaces 0
innodb_use_native_aio   ON
innodb_use_sys_malloc   ON
innodb_version  5.6.21
innodb_write_io_threads 4
insert_id   0
interactive_timeout 28800
join_buffer_size    262144
keep_files_on_create    OFF
key_buffer_size 16777216
key_cache_age_threshold 300
key_cache_block_size    1024
key_cache_division_limit    100
large_files_support ON
large_page_size 0
large_pages OFF
last_insert_id  0
lc_messages en_US
lc_messages_dir /rdsdbbin/mysql-5.6.21.R1/share/
lc_time_names   en_US
license GPL
local_infile    ON
lock_wait_timeout   31536000
locked_in_memory    OFF
log_bin ON
log_bin_basename    /rdsdbdata/log/binlog/mysql-bin-changelog
log_bin_index   /rdsdbdata/log/binlog/mysql-bin-changelog.index
log_bin_trust_function_creators OFF
log_bin_use_v1_row_events   OFF
log_error   /rdsdbdata/log/error/mysql-error.log
log_output  TABLE
log_queries_not_using_indexes   OFF
log_slave_updates   ON
log_slow_admin_statements   OFF
log_slow_slave_statements   OFF
log_throttle_queries_not_using_indexes  0
log_warnings    1
long_query_time 10.000000
low_priority_updates    OFF
lower_case_file_system  OFF
lower_case_table_names  0
master_info_repository  TABLE
master_verify_checksum  OFF
max_allowed_packet  4194304
max_binlog_cache_size   18446744073709547520
max_binlog_size 134217728
max_binlog_stmt_cache_size  18446744073709547520
max_connect_errors  100
max_connections 2540
max_delayed_threads 20
max_error_count 64
max_heap_table_size 16777216
max_insert_delayed_threads  20
max_join_size   18446744073709551615
max_length_for_sort_data    1024
max_prepared_stmt_count 16382
max_relay_log_size  0
max_seeks_for_key   18446744073709551615
max_sort_length 1024
max_sp_recursion_depth  0
max_tmp_tables  32
max_user_connections    0
max_write_lock_count    18446744073709551615
metadata_locks_cache_size   1024
metadata_locks_hash_instances   8
min_examined_row_limit  0
multi_range_count   256
myisam_data_pointer_size    6
myisam_max_sort_file_size   9223372036853727232
myisam_mmap_size    18446744073709551615
myisam_recover_options  OFF
myisam_repair_threads   1
myisam_sort_buffer_size 8388608
myisam_stats_method nulls_unequal
myisam_use_mmap OFF
net_buffer_length   16384
net_read_timeout    30
net_retry_count 10
net_write_timeout   60
new OFF
old OFF
old_alter_table OFF
old_passwords   0
open_files_limit    65535
optimizer_prune_level   1
optimizer_search_depth  62
optimizer_switch    index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,engine_condition_pushdown=on,index_condition_pushdown=on,mrr=on,mrr_cost_based=on,block_nested_loop=on,batched_key_access=off,materialization=on,semijoin=on,loosescan=on,firstmatch=on,subquery_materialization_cost_based=on,use_index_extensions=on
optimizer_trace enabled=off,one_line=off
optimizer_trace_features    greedy_search=on,range_optimizer=on,dynamic_range=on,repeated_subselect=on
optimizer_trace_limit   1
optimizer_trace_max_mem_size    16384
optimizer_trace_offset  -1
performance_schema  OFF
performance_schema_accounts_size    -1
performance_schema_digests_size -1
performance_schema_events_stages_history_long_size  -1
performance_schema_events_stages_history_size   -1
performance_schema_events_statements_history_long_size  -1
performance_schema_events_statements_history_size   -1
performance_schema_events_waits_history_long_size   -1
performance_schema_events_waits_history_size    -1
performance_schema_hosts_size   -1
performance_schema_max_cond_classes 80
performance_schema_max_cond_instances   -1
performance_schema_max_file_classes 50
performance_schema_max_file_handles 32768
performance_schema_max_file_instances   -1
performance_schema_max_mutex_classes    200
performance_schema_max_mutex_instances  -1
performance_schema_max_rwlock_classes   40
performance_schema_max_rwlock_instances -1
performance_schema_max_socket_classes   10
performance_schema_max_socket_instances -1
performance_schema_max_stage_classes    150
performance_schema_max_statement_classes    168
performance_schema_max_table_handles    -1
performance_schema_max_table_instances  -1
performance_schema_max_thread_classes   50
performance_schema_max_thread_instances -1
performance_schema_session_connect_attrs_size   -1
performance_schema_setup_actors_size    100
performance_schema_setup_objects_size   100
performance_schema_users_size   -1
pid_file    /rdsdbdata/log/mysql-3306.pid
plugin_dir  /rdsdbbin/mysql-5.6.21.R1/lib/plugin/
port    3306
preload_buffer_size 32768
profiling   OFF
profiling_history_size  15
protocol_version    10
proxy_user  
pseudo_slave_mode   OFF
pseudo_thread_id    22310
query_alloc_block_size  8192
query_cache_limit   1048576
query_cache_min_res_unit    4096
query_cache_size    1048576
query_cache_type    OFF
query_cache_wlock_invalidate    OFF
query_prealloc_size 8192
rand_seed1  0
rand_seed2  0
range_alloc_block_size  4096
read_buffer_size    262144
read_only   OFF
read_rnd_buffer_size    524288
relay_log   /rdsdbdata/log/relaylog/relaylog
relay_log_basename  /rdsdbdata/log/relaylog/relaylog
relay_log_index /rdsdbdata/log/relaylog/relaylog.index
relay_log_info_file relay-log.info
relay_log_info_repository   TABLE
relay_log_purge ON
relay_log_recovery  ON
relay_log_space_limit   0
report_host 
report_password 
report_port 3306
report_user 
rpl_stop_slave_timeout  31536000
secure_auth ON
secure_file_priv    /tmp/
server_id   507840347
server_id_bits  32
server_uuid 520896f1-1296-11e7-9164-0275a6150d51
simplified_binlog_gtid_recovery OFF
skip_external_locking   ON
skip_name_resolve   OFF
skip_networking OFF
skip_show_database  OFF
slave_allow_batching    OFF
slave_checkpoint_group  512
slave_checkpoint_period 300
slave_compressed_protocol   OFF
slave_exec_mode STRICT
slave_load_tmpdir   /rdsdbdata/tmp
slave_max_allowed_packet    1073741824
slave_net_timeout   3600
slave_parallel_workers  0
slave_pending_jobs_size_max 16777216
slave_rows_search_algorithms    TABLE_SCAN,INDEX_SCAN
slave_skip_errors   OFF
slave_sql_verify_checksum   ON
slave_transaction_retries   10
slave_type_conversions  
slow_launch_time    2
slow_query_log  OFF
slow_query_log_file /rdsdbdata/log/slowquery/mysql-slowquery.log
socket  /tmp/mysql.sock
sort_buffer_size    262144
sql_auto_is_null    OFF
sql_big_selects ON
sql_buffer_result   OFF
sql_log_bin ON
sql_log_off OFF
sql_mode    NO_ENGINE_SUBSTITUTION
sql_notes   ON
sql_quote_show_create   ON
sql_safe_updates    OFF
sql_select_limit    18446744073709551615
sql_slave_skip_counter  0
sql_warnings    OFF
ssl_ca  /rdsdbdata/rds-metadata/ca-cert.pem
ssl_capath  
ssl_cert    /rdsdbdata/rds-metadata/server-cert.pem
ssl_cipher  EXP1024-RC4-SHA:EXP1024-DES-CBC-SHA:AES256-SHA:AES128-SHA:DES-CBC3-SHA:DES-CBC-SHA:EXP-DES-CBC-SHA:EXP-RC2-CBC-MD5:RC4-SHA:RC4-MD5:EXP-RC4-MD5:NULL-SHA:NULL-MD5:DES-CBC3-MD5:DES-CBC-MD5:EXP-RC2-CBC-MD5:RC2-CBC-MD5:EXP-RC4-MD5:RC4-MD5:KRB5-DES-CBC3-MD5:KRB5-DES-CBC3-SHA:ADH-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:EDH-DSS-DES-CBC3-SHA:ADH-AES256-SHA:DHE-RSA-AES256-SHA:DHE-DSS-AES256-SHA:ADH-AES128-SHA:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA:EXP-KRB5-RC4-MD5:EXP-KRB5-RC2-CBC-MD5:EXP-KRB5-DES-CBC-MD5:KRB5-RC4-MD5:KRB5-DES-CBC-MD5:ADH-RC4-MD5:EXP-ADH-RC4-MD5:DHE-DSS-RC4-SHA:EXP1024-DHE-DSS-RC4-SHA:EXP1024-DHE-DSS-DES-CBC-SHA:EXP-KRB5-RC4-SHA:EXP-KRB5-RC2-CBC-SHA:EXP-KRB5-DES-CBC-SHA:KRB5-RC4-SHA:KRB5-DES-CBC-SHA:ADH-DES-CBC-SHA:EXP-ADH-DES-CBC-SHA:EDH-RSA-DES-CBC-SHA:EXP-EDH-RSA-DES-CBC-SHA:EDH-DSS-DES-CBC-SHA:EXP-EDH-DSS-DES-CBC-SHA
ssl_crl 
ssl_crlpath 
ssl_key /rdsdbdata/rds-metadata/server-key.pem
storage_engine  InnoDB
stored_program_cache    256
sync_binlog 1
sync_frm    ON
sync_master_info    10000
sync_relay_log  10000
sync_relay_log_info 10000
system_time_zone    UTC
table_definition_cache  1400
table_open_cache    2000
table_open_cache_instances  16
thread_cache_size   33
thread_concurrency  10
thread_handling one-thread-per-connection
thread_stack    262144
time_format %H:%i:%s
time_zone   UTC
timed_mutexes   OFF
timestamp   1490747772.419137
tmp_table_size  16777216
tmpdir  /rdsdbdata/tmp
transaction_alloc_block_size    8192
transaction_allow_batching  OFF
transaction_prealloc_size   4096
tx_isolation    REPEATABLE-READ
tx_read_only    OFF
unique_checks   ON
updatable_views_with_limit  YES
version 5.6.21-log
version_comment MySQL Community Server (GPL)
version_compile_machine x86_64
version_compile_os  Linux
wait_timeout    28800
warning_count   0
  • good to attache - SHOW VARIABLES as well – a_vlad Mar 28 '17 at 23:09
  • what I can suggest for start point - increase innodb_log files to 512M or 1024M, now it only 132Mb so if many big inserts this is could be reason. Do not put them huge, but 1-2G it must be ok – a_vlad Mar 29 '17 at 5:57
  • OK - I need to do a database failover by the looks of it to change that setting. I'll give it a whirl in a staging environment and see if the benchmark is any better. Thanks! – KJ Tsanaktsidis Mar 29 '17 at 22:17
1

Do you have Replication turned on? The first couple might be a cause for extra I/O (and CPU):

( Binlog_cache_disk_use ) = 268,012 / 83387 = 3.2 /sec -- Freq of binlog cache spilling to disk -- Increase binlog_cache_size.

( Binlog_cache_disk_use / Binlog_cache_use ) = 268,012 / 268284 = 99.9% -- Spill to disk -- Increase binlog_cache_size

( (Innodb_buffer_pool_reads + Innodb_buffer_pool_pages_flushed) ) = ((5484 + 32307952) ) / 83387 = 387 /sec -- InnoDB I/O -- Increase innodb_buffer_pool_size?

( Innodb_log_waits / Innodb_log_writes ) = 413 / 335941 = 0.12% -- Frequency of needing to wait to write to log -- Increase innodb_log_buffer_size.

( Uptime / 60 * innodb_log_file_size / Innodb_os_log_written ) = 83,387 / 60 * 128M / 534630468096 = 0.349 -- Minutes between InnoDB log rotations Beginning with 5.6.8, this can be changed dynamically; be sure to also change my.cnf. -- (The recommendation of 60 minutes between rotations is somewhat arbitrary.) Adjust innodb_log_file_size.

( ( Innodb_pages_read + Innodb_pages_written ) / Uptime / innodb_io_capacity ) = ( 5483 + 32308192 ) / 83387 / 200 = 193.8% -- If > 100%, need more io_capacity. -- Increase innodb_io_capacity if the drives can handle it. (AWS can probably handle it.)

( Select_scan / Com_select ) = 61,056 / 132852 = 46.0% -- % of selects doing full table scan. (May be fooled by Stored Routines.) -- Add indexes / optimize queries

( slave_pending_jobs_size_max / max_allowed_packet ) = 16M / 4M = 400.0% -- For parallel slave threads -- slave_pending_jobs_size_max must not be less than max_allowed_packet

( long_query_time ) = 10.000000 = 10 -- Cutoff (Seconds) for defining a "slow" query. -- Suggest 2

You have the Query Cache half-off. You should set both query_cache_type = OFF and query_cache_size = 0 . There is (according to a rumor) a 'bug' in the QC code that leaves some code on unless you turn off both of those settings.

Uptime is 23 hours. Were you running nothing but the benchmark for that whole time? I would prefer to analyze your workload, not a benchmark. (I usually find that benchmarks do not mirror reality.)

You asked about CPU... Some of the above issues relate directly to CPU; some relate indirectly. That is, the CPU is sometimes involved in things that otherwise seem like I/O. Perhaps a main example is when the code "spins the cpu" in hopes that a Mutex will be released or I/O will finished. The presumption is that that is 'faster' than hitting the OS.

In any case, address whatever issues you can, then we can make another analysis pass.

  • I made the following changes: Set binlog_cache_size to 130712 (from 32768), set innodb_io_capacity to 600 (from 200), set innodb_log_file_size to 1073741824 (from 134217728) and max_allowed_packet to 104857600 (from 52428800). There was no appreciable change in how my benchmark performed. – KJ Tsanaktsidis Apr 4 '17 at 3:36
  • I wouldn't' have a problem sharing our real production workload but we only get this kind of sustained BLOB inserts occasionally (depends on usage patterns). If it happens again I'd be happy to share. – KJ Tsanaktsidis Apr 4 '17 at 3:36
  • I'm leaning towards the conclusion that the problem is disk bandwidth - a db.r3.xlarge only has 500 Mbps of bandwidth to the SAN storage. I'm going to try turning on compression, otherwise we'll have to buy our way out of this with better disk I/O performance. – KJ Tsanaktsidis Apr 4 '17 at 3:38
  • InnoDB COMPRESSED compresses about 2:1. Manually (In the client) compressing ordinary English or code TEXT gets about 3:1 and cuts back on the bandwidth between the client and MySQL server. If practical, do the compression in the client. – Rick James Apr 4 '17 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.