Master DB Ran Out Of Space

Question

My master server ran out of space today (at 20:22). My transactions logs ended up broken into multiple files:

-rw-rw----  1 mysql mysql 228M Jul 31 20:22 mysql-bin.000914
-rw-rw----  1 mysql mysql  18K Jul 31 20:23 mysql-bin.000915
-rw-rw----  1 mysql mysql 1.5K Jul 31 20:24 mysql-bin.000916
-rw-rw----  1 mysql mysql 8.9K Jul 31 20:25 mysql-bin.000917
-rw-rw----  1 mysql mysql 1.2K Jul 31 20:25 mysql-bin.000918
-rw-rw----  1 mysql mysql 4.0K Jul 31 20:25 mysql-bin.000919
-rw-rw----  1 mysql mysql  982 Jul 31 20:26 mysql-bin.000920
-rw-rw----  1 mysql mysql 1.5K Jul 31 20:26 mysql-bin.000921
-rw-rw----  1 mysql mysql 2.1K Jul 31 20:26 mysql-bin.000922
-rw-rw----  1 mysql mysql  915 Jul 31 20:27 mysql-bin.000923
-rw-rw----  1 mysql mysql  17K Jul 31 20:28 mysql-bin.000924
-rw-rw----  1 mysql mysql  31K Jul 31 20:28 mysql-bin.000925
-rw-rw----  1 mysql mysql  54M Aug  1 02:00 mysql-bin.000926

This broke replication on my 4 slaves. Looking over other threads it seems I will have take a dump from the master and rebuild all my slaves. Is this correct?

Here's the slaves output.

Last_IO_Errno: 1595
Last_IO_Error: Relay log write failure: could not queue event from master
Master_Log_File: mysql-bin.000927
Read_Master_Log_Pos: 4
Relay_Log_File: mysql-relay-bin.000244
Relay_Log_Pos: 167
Relay_Master_Log_File: mysql-bin.000927

Skipping over the transactions that processed when it was out of space are fine. I tried advancing the position and file to the current file but that also didn't work.

stop slave;
change master to master_log_file='mysql-bin.000926', master_log_pos=0;
START SLAVE;
show slave status\G

which gave me:

Master_Log_File: mysql-bin.000926
Read_Master_Log_Pos: 4
Relay_Log_File: mysql-relay-bin.000001
Relay_Log_Pos: 4
Relay_Master_Log_File: mysql-bin.000926
Last_IO_Errno: 1236
Last_IO_Error: Got fatal error 1236 from master when reading data from binary log: 'Binary log is not open'

I'd prefer not having to rebuild my slaves if possible. Thanks.

Answer 1

This isn't an exact answer because I don't know how I did resolved the issue but the following options did resolve my issue. When I tried to create a mysql dump to re-build the server I got:

mysqldump: Error: Binlogging on server not active

Which to me indicated there was something wrong with the mysql configuration on the master because binary logging is enabled in the my.cnf. I also looked at the most recent binary log file for the past 6 hours and it was only 2.8K. I ran

service mysql restart

which got my binary logs working as intended. The new binary log began functioning again as expected (within 5 minutes 2.8M). So now my log file became:

-rw-rw----  1 mysql mysql  54M Aug  1 02:00 mysql-bin.000926
-rw-rw----  1 mysql mysql 2.8K Aug  1 10:58 mysql-bin.000927
-rw-rw----  1 mysql mysql 2.4M Aug  1 11:02 mysql-bin.000928

I then went back to my slaves and re-ran the change master commands to the new bin log and it worked as expected.

stop slave;
change master to master_log_file='mysql-bin.000928', master_log_pos=0;
START SLAVE;

With this show slave status\G now shows:

Master_Log_File: mysql-bin.000928
Read_Master_Log_Pos: 2439078
Relay_Log_File: mysql-relay-bin.000002
Relay_Log_Pos: 299660
Relay_Master_Log_File: mysql-bin.000928
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
....
Last_IO_Errno: 0
Last_IO_Error: 
Last_SQL_Errno: 0
Last_SQL_Error: