I have a mysql database instance deployed in a multi-az configuration on Amazon RDS. We've seen replication fail a couple of times in the last two months, with an error message like,
Apply Error 1590: The incident LOST_EVENTS occured on the master. Message: error writing to the binary log
Looking closer into the logs on the replica, I noticed that it was trying to execute an event (we have a heartbeat event that is
ENABLED on the primary, and should be in a
SLAVESIDE_DISABLED state on the replica(s). Looking closer, I realized the failing replica has this event in
ENABLED state, and since the server is running as ``--read-only` it fails while trying to run the event. There are errors like below in the replica logs.
2016-02-28 20:55:00 16040 [ERROR] Event Scheduler: [database_name@%][database_name.update_heartbeat_event] The MySQL server is running with the --read-only option so it cannot execute this statement
I tried dropping the event on the primary, but it had no effect on the replica, presumably because replication is in failing at this point.
Any pointers to help fix this (besides creating a new readonly replica from the primary?)