MySQL master binlog corruption

Question

We recently made some changes to our infrastructure and now I can't keep MySQL replication running. The slave complains of a corrupt binlog and resetting it doesn't help. I keep seeing entries like this in my master log:

BINLOG '
WFxKTRNJAAAAPwAAAKY/YwAAABsAAAAAAAEACHdlYmVkaTMwAA1QYXJ0bmVyQ29uZmlnAAQICA8P
BC0AYwAA
WFxKTRhJAAAAXAAAAAJAYwAAABsAAAAAAAEABP//8BoSAAAAAAAAggMAAAAAAAAJUEFDS0NPVU5U
Azc3MfAaEgAAAAAAAIIDAAAAAAAACVBBQ0tDT1VOVAM3NzI=
'/*!*/;

In the slave log, this just comes across as "Unknown Event" and the replication fails at that position.

Here is my my.cnf for the master:

[mysqld]
datadir=/data/mysql-data
socket=/tmp/mysql.sock
user=mysql
log-error=/var/log/mysqld.log

binlog-do-db    = db1
binlog-do-db    = db2
binlog-do-db    = db3
binlog-do-db    = db4
binlog_format   = 'MIXED'

log-bin = /data/mysql-binlogs/mysql-bin.log
server-id=73
report-host=thisserver.mydomain.com

thread_cache_size = 30
key_buffer_size = 700M
myisam_sort_buffer_size = 300M
table_cache = 256
sort_buffer_size = 4M
read_buffer_size = 1M
innodb_data_home_dir            = /data/mysql-data/
innodb_data_file_path           = InnoDB:100M:autoextend
set-variable                    = innodb_buffer_pool_size=500M
set-variable                    = innodb_additional_mem_pool_size=10M
set-variable                    = max_connections=500
innodb_log_group_home_dir       = /data/mysql-data
set-variable                    = innodb_log_file_size=20M
set-variable                    = innodb_log_buffer_size=8M
innodb_flush_log_at_trx_commit  = 1

Does anybody know what might be causing the gibberish in my master binlogs?

Answer 1

Surprisingly, that's not gibberish.

That indeed appears at the top of binlogs whenever you do mysqlbinlog to a binary log generated using MySQL 5.1 and MySQL 5.5. You will not see that gibberish in binary logs for MySQL 5.0 and back.

This is why the start point for replication from an empty binary log is

• 107 for MySQL 5.5
• 106 for MySQL 5.1
• 98 for MySQL 5.0 and back

This is good to remember if you do MySQL Replication where the Master if MySQL 5.1 and the slave is MySQL 5.0. This could present a really big headache.

Replication from Master using 5.0 and Slave using 5.1 works fine, not the other way around.(According to MySQL Documentation, it is generally not supported for 3 reasons: 1) Binary Log Format, 2) Row-based Replication, 3) SQL Incompatibility).

Anyway, do a mysqlbinlog on the offending binary log on the master. If the resulting dump produces gibberish in the middle of the dump (which I have seen a couple of times in my DBA career) you may have to skip to position 98 (MySQL 5.0) or 106 (MySQL 5.1) or 107 (MySQL 5.5) of the master's next binary log and start replicating from there (SOB :( you may need to use MAATKIT tools mk-table-checksum and mk-table-sync to reload master changes not on the slave [if you want to be a hero]; even worse, mysqldump the master and reload the slave and start replication totally over [if you don't want to be a hero])

If the mysqlbinlog of the master is completely readable after the top gibberish you saw, it is possible the master's binary log is fine but the relay log on the slave is corrupt (due to transmission/CRC errors). If that's the case, just reload the relay logs by issuing the CHANGE MASTER TO command as follows:

STOP SLAVE;
CHANGE MASTER TO
MASTER_HOST='< master-host ip or DNS >',
MASTER_PORT=3306,
MASTER_USER='< usernmae >',
MASTER_PASSWORD='< password >',
MASTER_LOG_FILE='< MMMM >',
MASTER_LOG_POS=< PPPP >;
START SLAVE;

Where

• MMMM is the last file used from the Master that was last processed on the Slave
• PPPP is the last position used from the Master that was last processed on the Slave

You can get MMMM and PPPP by doing SHOW SLAVE STATUS\G and using

• Relay_Master_Log_File for MMMM
• Exec_Master_Log_Pos for PPPP

Try it out and let me know !!!

BTW running CHANGE MASTER TO command erases the slave's current relay logs and starts fresh.