Max allowed packet changed still replication not starting

Question

Suddenly I notice my replication stopped and first I got some primary key duplication error when I checked using the show slave status\G. Then I remove those lines and restarted and ending up now with this error. So I first increase both the master and slave max_allowed_packet to 1G yet not working and now I changed to 2G yet is the same. I have run stop slave and change master to master_host='192.168.**',master_user='repu1',master_password='***', master_log_file='mysql-bin.000039', master_log_pos=640109146; start slave; yet I keep getting the same error. "Got fatal error 1236 from master when reading data from binary log: 'log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master'"

I have checked the repu1 password using this command mysql -h 192.168.* -u repu1 -p its working fine too.

show slave status \G;
*************************** 1. row ***************************
               Slave_IO_State:
                  Master_Host: 192.168.100.12
                  Master_User: repu1
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000039
          Read_Master_Log_Pos: 640355070
               Relay_Log_File: mysqld-relay-bin.000002
                Relay_Log_Pos: 251
        Relay_Master_Log_File: mysql-bin.000039
             Slave_IO_Running: No
            Slave_SQL_Running: No
              Replicate_Do_DB: dbn
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1594
                   Last_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary log is corrupted (you can check this by running 'mysqlbinlog' on the binary log), the slave's relay log is corrupted (you can check this by running 'mysqlbinlog' on the relay log), a network problem, or a bug in the master's or slave's MySQL code. If you want to check the master's binary log or slave's relay log, you will be able to know their names by issuing 'SHOW SLAVE STATUS' on this slave.
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 640109146
              Relay_Log_Space: 247058
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 1236
                Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'log event entry exceeded max_allowed_packet; Increase max_allowed_packet on master'
               Last_SQL_Errno: 1594
               Last_SQL_Error: Relay log read failure: Could not parse relay log event entry. The possible reasons are: the master's binary lo

Answer 1

Try these steps:

1) Take a mysqldump from master

2) Restore it on a slave, whether it gets restored successfully?

3) You should not keep max_allowed_packet to a high value like 1G or 2G keep it minimal like 16M or 32M and restart Master server.

4) Apply same max_allowed_packet size in slave as well.3

5) Now, stop the slave, execute Change Master .... query accourdingly and start slave

I hope this will work for you.

Answer 2

This error is usually caused by a hardware glitch or a bug so obscure that it hasn't been found in over a decade. The optimal solution that will definitely fix it without any risk of anomalies creeping in is to fully re-initialize the replica. You can do this without downtime or locking by using either xtrabackup or mysqldump --single-transaction.

Having said that, this can be prohibitively expensive if you have terabytes of data, so there may be a way you can restore replication without rebuilding the slave completely. Use mysqlbinlog on the master to identify where the transactions begin, and look for the adjacent transactions near position 640109146 (your Exec_Master_Log_Pos). There is a very good chance that the position listed is actually erroneous and no transaction in the binlog starts at that position. You are also very likely to find that the coordinate distances between these transactions are nowhere near your max_allowed_packet value.

If that is the case, you can reset your replication coordinates on the slave to the last transaction mysqlbinlog lists that is before the position the slave thinks it is at. You can find more details on exactly what to do in this mysql troubleshooting article. There is risk associated with doing this (if the coordinates got corrupted, there is no telling what else got corrupted), but if doing the full re-initialisation immediately is prohibitively expensive, you might be able to buy yourself some time using this approach, at least until you can verify (e.g. using pt-table-checksum whether you have to rebuild the entire replica data set.

HTH

Disclosure: I wrote the referenced article.