0

I am running a migration script which copies a large amount of data from one table into another and the process is now running for more than 24 hours and failing, as well as generating alarms from the AWS Cloudwatch monitor regarding high IOPS and through-put.

Server version: 10.2.12-MariaDB-log MariaDB Server

I also see these errors:

2019-04-23 16:09:44 47291578124032 [Warning] Aborted connection 17189 to db: 'mydb' user: 'APPDBA' host: '1.2.3.4' (The table 'modification' is full)
2019-04-23 16:09:44 47291578124032 [ERROR] WSREP: failed to initialize io-cache
2019-04-23 16:09:44 47291578124032 [Warning] WSREP: binlog trx cache not empty (0 bytes) @ connection close 17189
2019-04-23 16:09:44 47291578124032 [Warning] WSREP: binlog stmt cache not empty (0 bytes) @ connection close 17189

and (although I've no idea if it's connected - it is much later):

2019-04-24 13:33:24 47292002281216 [Warning] 
Aborted connection 18175 to db: 'mydb' user: 'service_user_a' host: '1.2.3.4' (Got an error reading communication packets)

These are the tables and the data copy SQL:

CREATE TABLE old_mod (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT,
  userId INT UNSIGNED NOT NULL,
  epochTime BIGINT NOT NULL,
  forecastId INT UNSIGNED NOT NULL,
  description CHAR(12) NOT NULL,
  auxText TEXT NULL,
  auxDate DATE NULL,
  CONSTRAINT pk_modification PRIMARY KEY (id),
  CONSTRAINT fk_modification_forecastId FOREIGN KEY(forecastId)
    REFERENCES forecast(id),
  CONSTRAINT fk_modification_userId FOREIGN KEY (userId)
    REFERENCES user(id)
) ENGINE=InnoDB
  MAX_ROWS 12345678;

CREATE INDEX ix_modification_query_by_user_and_forecast
  ON old_mod (userId, forecastId, description, auxDate, epochTime);

CREATE INDEX ix_modification_last_save_query
  ON old_mod (forecastId, description, auxDate);

I drop the indices on the original table before creating the new one, so that I can re-use the index names - I didn't think that would slow it down - or does it?

In my operation, originally the table above is modification but I rename it to old_mod. I would have just altered it in place, but I kept getting other errors - see here if interested: MariaDB "ERROR 1034 (HY000) at line 41: Index for table 'xyz' is corrupt" on importing dump

The operation then creates this table and inserts into it:

CREATE TABLE modification (
  id INT UNSIGNED NOT NULL AUTO_INCREMENT,
  username CHAR(30) NOT NULL,
  display_name VARCHAR(255) NULL,
  epoch_time BIGINT NOT NULL,
  forecast_id INT UNSIGNED NOT NULL,
  forecast_symbol VARCHAR(255) NOT NULL,
  group_id INT UNSIGNED NOT NULL,
  group_name CHAR(100) NOT NULL,
  description CHAR(12) NOT NULL,
  aux_text TEXT NULL,
  aux_date DATE NULL,
  CONSTRAINT pk_modification PRIMARY KEY (id),
  KEY ix_modification_query_by_user_and_forecast
    (username, forecast_id, description, aux_date, epoch_time),
  KEY ix_modification_last_save_query
    (forecast_id, description, aux_date),
  KEY ix_modification_forecast_symbol (forecast_symbol)
) ENGINE=InnoDB
  DEFAULT CHARSET=utf8mb4
  COLLATE=utf8mb4_unicode_ci
  MAX_ROWS 500123123;


-- add username to modification via userid
INSERT INTO modification (username, epoch_time, forecast_id, forecast_symbol,
                          group_id, group_name,
                          description, aux_text, aux_date)
    SELECT u.name, m.epochTime, m.forecastId, f.name, 1, 'Not recorded',
           m.description, m.auxText, m.auxDate
    FROM old_mod m
    LEFT JOIN `user` u ON u.id = m.userId
    LEFT JOIN forecast f ON f.id = m.forecastId;

I'm not sure what config I need to adjust, or whether I can improve my table schema.

My config:

aria_block_size                                   8192
aria_checkpoint_interval                          30
aria_checkpoint_log_activity                      1048576
aria_encrypt_tables                               OFF
aria_force_start_after_recovery_failures          0
aria_group_commit                                 none
aria_group_commit_interval                        0
aria_log_file_size                                1073741824
aria_log_purge_type                               immediate
aria_max_sort_file_size                           9223372036853727232
aria_page_checksum                                ON
aria_pagecache_age_threshold                      300
aria_pagecache_buffer_size                        134217728
aria_pagecache_division_limit                     100
aria_pagecache_file_hash_size                     512
aria_recover_options                              BACKUP,QUICK
aria_repair_threads                               1
aria_sort_buffer_size                             268434432
aria_stats_method                                 nulls_unequal
aria_sync_log_dir                                 NEWFILE
aria_used_for_temp_tables                         ON
auto_increment_increment                          1
auto_increment_offset                             1
autocommit                                        ON
automatic_sp_privileges                           ON
back_log                                          558
basedir                                           /rdsdbbin/mysql/
big_tables                                        OFF
binlog_annotate_row_events                        ON
binlog_cache_size                                 32768
binlog_checksum                                   CRC32
binlog_commit_wait_count                          0
binlog_commit_wait_usec                           100000
binlog_direct_non_transactional_updates           OFF
binlog_format                                     MIXED
binlog_optimize_thread_scheduling                 ON
binlog_row_image                                  FULL
binlog_stmt_cache_size                            32768
bulk_insert_buffer_size                           8388608
check_constraint_checks                           ON
collation_connection                              latin1_swedish_ci
collation_database                                latin1_swedish_ci
collation_server                                  latin1_swedish_ci
completion_type                                   NO_CHAIN
concurrent_insert                                 AUTO
connect_timeout                                   10
datadir                                           /rdsdbdata/db/
default_storage_engine                            InnoDB
default_tmp_storage_engine
default_week_format                               0
delay_key_write                                   ON
delayed_insert_limit                              100
delayed_insert_timeout                            300
delayed_queue_size                                1000
div_precision_increment                           4
encrypt_binlog                                    OFF
encrypt_tmp_disk_tables                           OFF
encrypt_tmp_files                                 OFF
enforce_storage_engine
event_scheduler                                   OFF
expensive_subquery_limit                          100
expire_logs_days                                  0
explicit_defaults_for_timestamp                   OFF
extra_max_connections                             1
extra_port                                        0
flush                                             OFF
flush_time                                        0
foreign_key_checks                                ON
general_log                                       OFF
general_log_file                                  /rdsdbdata/log/general/mysql-general.log
group_concat_max_len                              1048576
have_compress                                     YES
have_crypt                                        YES
have_dynamic_loading                              YES
have_geometry                                     YES
have_openssl                                      YES
have_profiling                                    YES
have_query_cache                                  YES
have_rtree_keys                                   YES
have_ssl                                          YES
have_symlink                                      YES
histogram_size                                    0
histogram_type                                    SINGLE_PREC_HB
host_cache_size                                   730
hostname                                          ip-172-20-2-15
ignore_builtin_innodb                             OFF
ignore_db_dirs
init_connect
init_file
init_slave
innodb_adaptive_flushing                          ON
innodb_adaptive_flushing_lwm                      10.000000
innodb_adaptive_hash_index                        ON
innodb_adaptive_hash_index_partitions             8
innodb_adaptive_hash_index_parts                  8
innodb_adaptive_max_sleep_delay                   150000
innodb_autoextend_increment                       64
innodb_autoinc_lock_mode                          1
innodb_background_scrub_data_check_interval       3600
innodb_background_scrub_data_compressed           OFF
innodb_background_scrub_data_interval             604800
innodb_background_scrub_data_uncompressed         OFF
innodb_buf_dump_status_frequency                  0
innodb_buffer_pool_chunk_size                     134217728
innodb_buffer_pool_dump_at_shutdown               ON
innodb_buffer_pool_dump_now                       OFF
innodb_buffer_pool_dump_pct                       25
innodb_buffer_pool_filename                       ib_buffer_pool
innodb_buffer_pool_instances                      8
innodb_buffer_pool_load_abort                     OFF
innodb_buffer_pool_load_at_startup                ON
innodb_buffer_pool_load_now                       OFF
innodb_buffer_pool_populate                       OFF
innodb_buffer_pool_size                           24696061952
innodb_change_buffer_max_size                     25
innodb_change_buffering                           all
innodb_checksum_algorithm                         crc32
innodb_checksums                                  ON
innodb_cleaner_lsn_age_factor                     DEPRECATED
innodb_cmp_per_index_enabled                      OFF
innodb_commit_concurrency                         0
innodb_compression_algorithm                      zlib
innodb_compression_default                        OFF
innodb_compression_failure_threshold_pct          5
innodb_compression_level                          6
innodb_compression_pad_pct_max                    50
innodb_concurrency_tickets                        5000
innodb_corrupt_table_action                       deprecated
innodb_data_file_path                             ibdata1:12M:autoextend
innodb_data_home_dir                              /rdsdbdata/db/innodb
innodb_deadlock_detect                            ON
innodb_default_encryption_key_id                  1
innodb_default_row_format                         dynamic
innodb_defragment                                 OFF
innodb_disable_sort_file_cache                    OFF
innodb_disallow_writes                            OFF
innodb_doublewrite                                ON
innodb_empty_free_list_algorithm                  DEPRECATED
innodb_encrypt_log                                OFF
innodb_encrypt_tables                             OFF
innodb_encryption_rotate_key_age                  1
innodb_encryption_rotation_iops                   100
innodb_encryption_threads                         0
innodb_fake_changes                               OFF
innodb_fast_shutdown                              1
innodb_fatal_semaphore_wait_threshold             600
innodb_file_format                                Barracuda
innodb_file_format_check                          ON
innodb_file_format_max                            Barracuda
innodb_file_per_table                             ON
innodb_fill_factor                                100
innodb_flush_log_at_timeout                       1
innodb_flush_log_at_trx_commit                    0
innodb_flush_method                               O_DIRECT
innodb_flush_neighbors                            1
innodb_flush_sync                                 ON
innodb_flushing_avg_loops                         30
innodb_force_load_corrupted                       OFF
innodb_force_primary_key                          OFF
innodb_force_recovery                             0
innodb_foreground_preflush                        DEPRECATED
innodb_ft_aux_table
innodb_ft_cache_size                              8000000
innodb_ft_enable_diag_print                       OFF
innodb_ft_enable_stopword                         ON
innodb_ft_max_token_size                          84
innodb_ft_min_token_size                          3
innodb_ft_num_word_optimize                       2000
innodb_ft_result_cache_limit                      2000000000
innodb_ft_server_stopword_table
innodb_ft_sort_pll_degree                         2
innodb_ft_total_cache_size                        640000000
innodb_ft_user_stopword_table
innodb_idle_flush_pct                             100
innodb_immediate_scrub_data_uncompressed          OFF
innodb_instrument_semaphores                      OFF
innodb_io_capacity                                200
innodb_io_capacity_max                            2000
innodb_kill_idle_transaction                      0
innodb_large_prefix                               ON
innodb_lock_schedule_algorithm                    vats
innodb_lock_wait_timeout                          50
innodb_locking_fake_changes                       OFF
innodb_locks_unsafe_for_binlog                    OFF
innodb_log_arch_dir
innodb_log_arch_expire_sec                        0
innodb_log_archive                                OFF
innodb_log_block_size                             0
innodb_log_buffer_size                            8388608
innodb_log_checksum_algorithm                     DEPRECATED
innodb_log_checksums                              ON
innodb_log_compressed_pages                       ON
innodb_log_file_size                              134217728
innodb_log_files_in_group                         2
innodb_log_group_home_dir                         /rdsdbdata/log/innodb
innodb_log_write_ahead_size                       8192
innodb_lru_scan_depth                             1024
innodb_max_bitmap_file_size                       0
innodb_max_changed_pages                          0
innodb_max_dirty_pages_pct                        75.000000
innodb_max_dirty_pages_pct_lwm                    0.000000
innodb_max_purge_lag                              0
innodb_max_purge_lag_delay                        0
innodb_max_undo_log_size                          10485760
innodb_mirrored_log_groups                        0
innodb_monitor_disable
innodb_monitor_enable
innodb_monitor_reset
innodb_monitor_reset_all
innodb_mtflush_threads                            8
innodb_numa_interleave                            OFF
innodb_old_blocks_pct                             37
innodb_old_blocks_time                            1000
innodb_online_alter_log_max_size                  134217728
innodb_open_files                                 2000
innodb_optimize_fulltext_only                     OFF
innodb_page_cleaners                              4
innodb_page_size                                  16384
innodb_prefix_index_cluster_optimization          OFF
innodb_purge_batch_size                           300
innodb_purge_rseg_truncate_frequency              128
innodb_purge_threads                              4
innodb_random_read_ahead                          OFF
innodb_read_ahead_threshold                       56
innodb_read_io_threads                            4
innodb_read_only                                  OFF
innodb_replication_delay                          0
innodb_rollback_on_timeout                        OFF
innodb_rollback_segments                          128
innodb_sched_priority_cleaner                     0
innodb_scrub_log                                  OFF
innodb_scrub_log_speed                            256
innodb_show_locks_held                            0
innodb_show_verbose_locks                         0
innodb_sort_buffer_size                           1048576
innodb_spin_wait_delay                            6
innodb_status_output                              OFF
innodb_status_output_locks                        OFF
innodb_strict_mode                                ON
innodb_support_xa                                 ON
innodb_sync_array_size                            1
innodb_sync_spin_loops                            30
innodb_table_locks                                ON
innodb_temp_data_file_path                        ibtmp1:12M:autoextend
innodb_thread_concurrency                         0
innodb_thread_sleep_delay                         10000
innodb_tmpdir
innodb_track_changed_pages                        OFF
innodb_track_redo_log_now                         OFF
innodb_undo_directory                             ./
innodb_undo_log_truncate                          OFF
innodb_undo_logs                                  128
innodb_undo_tablespaces                           0
innodb_use_atomic_writes                          ON
innodb_use_fallocate                              OFF
innodb_use_global_flush_log_at_trx_commit         OFF
innodb_use_mtflush                                OFF
innodb_use_native_aio                             ON
innodb_use_stacktrace                             OFF
innodb_use_trim                                   ON
innodb_version                                    5.7.20
innodb_write_io_threads                           4
interactive_timeout                               28800
join_buffer_size                                  262144
join_buffer_space_limit                           2097152
join_cache_level                                  2
keep_files_on_create                              OFF
key_buffer_size                                   16777216
key_cache_age_threshold                           300
key_cache_block_size                              1024
key_cache_division_limit                          100
key_cache_file_hash_size                          512
key_cache_segments                                0
large_files_support                               ON
large_page_size                                   0
large_pages                                       OFF
local_infile                                      ON
lock_wait_timeout                                 86400
locked_in_memory                                  OFF
log_bin                                           ON
log_bin_basename                                  /rdsdbdata/log/binlog/mysql-bin-changelog
log_bin_compress                                  OFF
log_bin_compress_min_len                          256
log_bin_index                                     /rdsdbdata/log/binlog/mysql-bin-changelog.index
log_bin_trust_function_creators                   ON
log_error                                         /rdsdbdata/log/error/mysql-error.log
long_query_time                                   10.000000
low_priority_updates                              OFF
lower_case_file_system                            OFF
lower_case_table_names                            0
master_verify_checksum                            OFF
max_allowed_packet                                562036736
max_binlog_cache_size                             18446744073709547520
max_binlog_size                                   134217728
max_binlog_stmt_cache_size                        18446744073709547520
max_connect_errors                                100
max_connections                                   2540
max_delayed_threads                               20
max_digest_length                                 1024
max_error_count                                   64
max_heap_table_size                               16777216
max_insert_delayed_threads                        20
max_join_size                                     18446744073709551615
max_length_for_sort_data                          1024
max_long_data_size                                16777216
max_prepared_stmt_count                           16382
max_recursive_iterations                          4294967295
max_relay_log_size                                134217728
max_seeks_for_key                                 4294967295
max_session_mem_used                              9223372036854775807
max_sort_length                                   1024
max_sp_recursion_depth                            0
max_statement_time                                0.000000
max_tmp_tables                                    32
max_user_connections                              0
max_write_lock_count                              4294967295
metadata_locks_cache_size                         1024
metadata_locks_hash_instances                     8
min_examined_row_limit                            0
mrr_buffer_size                                   262144
multi_range_count                                 256
mysql56_temporal_format                           ON
net_buffer_length                                 16384
net_read_timeout                                  30
net_retry_count                                   10
net_write_timeout                                 60
open_files_limit                                  65535
optimizer_prune_level                             1
optimizer_search_depth                            62
optimizer_selectivity_sampling_limit              100
optimizer_switch                                  index_merge=on,index_merge_union=on,index_merge_sort_union=on,index_merge_intersection=on,index_merge_sort_intersection=off,engine_condition_pushdown=off,index_condition_pushdown=on,de
optimizer_use_condition_selectivity               1
performance_schema                                OFF
pid_file                                          /rdsdbdata/log/mysql-3306.pid
plugin_dir                                        /rdsdbbin/mariadb-10.2.12.R1/lib/plugin/
plugin_maturity                                   unknown
port                                              3306
preload_buffer_size                               32768
profiling                                         OFF
profiling_history_size                            15
progress_report_time                              5
protocol_version                                  10
query_alloc_block_size                            16384
query_cache_limit                                 1048576
query_cache_min_res_unit                          4096
query_cache_size                                  1048576
query_cache_strip_comments                        OFF
query_cache_type                                  OFF
query_cache_wlock_invalidate                      OFF
query_prealloc_size                               24576
range_alloc_block_size                            4096
read_binlog_speed_limit                           0
read_buffer_size                                  262144
read_only                                         OFF
read_rnd_buffer_size                              524288
rowid_merge_buff_size                             8388608
secure_auth                                       ON
secure_file_priv                                  /tmp/
server_id                                         1853500884
session_track_schema                              ON
session_track_state_change                        OFF
session_track_system_variables
session_track_transaction_info                    OFF
skip_external_locking                             ON
skip_name_resolve                                 OFF
skip_networking                                   OFF
skip_show_database                                OFF
slow_launch_time                                  2
slow_query_log                                    OFF
slow_query_log_file                               /rdsdbdata/log/slowquery/mysql-slowquery.log
socket                                            /tmp/mysql.sock
sort_buffer_size                                  2097152
sql_auto_is_null                                  OFF
sql_big_selects                                   ON
sql_buffer_result                                 OFF
sql_log_bin                                       ON
sql_log_off                                       OFF
sql_safe_updates                                  OFF
sql_select_limit                                  18446744073709551615
sql_slave_skip_counter                            0
sql_warnings                                      OFF
standard_compliant_cte                            ON
storage_engine                                    InnoDB
stored_program_cache                              256
strict_password_validation                        ON
table_definition_cache                            400
table_open_cache                                  2000
table_open_cache_instances                        8
thread_cache_size                                 256
thread_concurrency                                10
thread_handling                                   one-thread-per-connection
thread_pool_idle_timeout                          60
thread_pool_max_threads                           65536
thread_pool_oversubscribe                         3
thread_pool_prio_kickup_timer                     1000
thread_pool_priority                              auto
thread_pool_size                                  4
thread_pool_stall_limit                           500
thread_stack                                      262144
tmp_disk_table_size                               18446744073709551615
tmp_memory_table_size                             16777216
tmp_table_size                                    16777216
tmpdir                                            /rdsdbdata/tmp
transaction_alloc_block_size                      8192
transaction_prealloc_size                         4096
tx_isolation                                      REPEATABLE-READ
tx_read_only                                      OFF
unique_checks                                     ON
updatable_views_with_limit                        YES
use_stat_tables                                   NEVER
userstat                                          OFF
version                                           10.2.12-MariaDB-log
wait_timeout                                      28800

Update: here's the query plan.

| id   | select_type | table | type   | possible_keys | key     | key_len | ref                 | rows   | filtered | Extra |
+------+-------------+-------+--------+---------------+---------+---------+---------------------+--------+----------+-------+
|    1 | SIMPLE      | m     | ALL    | NULL          | NULL    | NULL    | NULL                | 441833 |   100.00 |       |
|    1 | SIMPLE      | u     | eq_ref | PRIMARY       | PRIMARY | 4       | myDb.m.userId     |      1 |   100.00 |       |
|    1 | SIMPLE      | f     | eq_ref | PRIMARY       | PRIMARY | 4       | myDb.m.forecastId |      1 |   100.00 |       |
  • For starters, if you know the data is good, I'd probably create the indexes after the load (except for the PK) – Philᵀᴹ Apr 24 at 19:29
  • Oh, you say you drop all of the indexes on the table you're inserting from? That sounds like a terrible idea, given you're using it for populating the new table & are joining with another table. You'll probably want one on forecastId and userId, or maybe both. How many rows are in the table you're joining with? Can you show us a query plan for the insert? – Philᵀᴹ Apr 24 at 19:33
  • ??? There is NO userId field in old_mod table. How can you use it in JOIN??? – Akina Apr 25 at 4:59
  • @Akina it was a typo from reducing the replication script down to the relevant statements. I've corrected the question. – Adam Apr 25 at 13:02
  • OK, I've added the explain plan. I thought that the indices on the old_mod table were unnecessary because the query is inserting every row from the table anyway, i.e. it's doing a table scan. Or is that a massive oversimplification? – Adam Apr 25 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.