0

Since we have upgraded from 5.7 to 8.0 (8.0.30-22), the servers don’t appear to be performing as well with the replication. The will frequently show 1 second repl lag, sometimes shooting up to 30 seconds. With the 5.7 servers we were running a few days ago, repl lag would rarely go above 1. CPU usage is generally higher than before. Our configuration is almost identical to what it was on 5.7 so I am unsure what to check here.

Servers: 1 x Write master and 5 x Read slave, all on Rackspace 15GB IO servers, with the database stored on a 150GB RAID 10-protected SSD. Database is around 50GB.

my.cnf settings (slave):

user = mysql
pid-file = /var/run/mysqld/mysqld.pid
socket = /var/run/mysqld/mysqld.sock
port = 3306
basedir = /usr
datadir = /data/var/lib/mysql
tmpdir = /tmp
lc-messages-dir = /usr/share/mysql
skip-external-locking
sql-mode = ‘STRICT_TRANS_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,NO_ENGINE_SUBSTITUTION’

bind-address = x.x.x.x
key_buffer_size = 16M
max_allowed_packet = 16M
lower_case_table_names = 1
thread_stack = 192K
thread_cache_size = 8

max_connections = 525
max-connect-errors = 1000000
skip-name-resolve
log_error = /var/log/mysql/error.log

server-id = 1670426130
disable_log_bin

innodb_file_per_table
innodb_buffer_pool_size = 8G
innodb_flush_log_at_trx_commit = 2
sync_binlog = 0
innodb_flush_method = O_DIRECT
slave_compressed_protocol = 1
innodb_io_capacity_max=4000
innodb_io_capacity=400
innodb_log_file_size = 128M
slave_type_conversions = ALL_NON_LOSSY

relay_log = /var/lib/mysql-relay-logs/mysql-relay-bin
relay_log_index = /var/lib/mysql-relay-logs/mysql-relay-index

iostat output (xvde is the data drive):

Device             tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn    kB_dscd
xvda              2.39        43.10        12.91         0.00    3611678    1081510          0
xvdd              0.20         8.58         0.00         0.00     718620          0          0
xvde            183.25       123.03      3116.60         0.00   10309873  261177252          0

Output from show slave status:

*************************** 1. row ***************************
  *************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: x.x.x.x
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.013007
          Read_Master_Log_Pos: 5121445
               Relay_Log_File: mysql-relay-bin.025340
                Relay_Log_Pos: 218802
        Relay_Master_Log_File: mysql-bin.013007
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 218586
              Relay_Log_Space: 5121918
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 12
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 1670425806
                  Master_UUID: 9ac7164c-7625-11ed-8536-bc764e080bec
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Waiting for dependent transaction to commit
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.01 sec)

Any help would be appreciated.

1
  • Set long_query_time = 1 and see what shows up in the slowlog.
    – Rick James
    Jan 18 at 3:04

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.