I have an AWS RDS Mysql master with 1 AWS RDS slave and 1 ec2 mysql slave replicating this master instance. I already set the binlog retention hours to 7 days with the command:
CALL mysql.rds_set_configuration('binlog retention hours', 168);
I checked the value and verifed that it is set to 168 by command
I have to stop ec2 mysql slave for 8-10 hours a day and starting this slave failed with
Slave I/O for channel '': Got fatal error 1236 from master when reading data from binary log: 'Could not find first log file name in binary log index file', Error_code: 1236
After I got this error on ec2 mysql slave during startup, I checked the master instance's BinLogDiskUsage cloudwatch metric. This metric dramatically drops to near zero value even the "binlog retention hours" is set to 168 hours. Also I can see that binary logs file count is dropped from more than hundred files to 5-6 files at that time.
As per documentation, setting "binlog retention hours" to 168 hours guarantees that the binary logs are kept for 7 days. Is there any other option that may reset/override this setting for a moment or any action taken on DB instance can cause of that binary logs files to be purged?