0

I have a mysql database instance deployed in a multi-az configuration on Amazon RDS. We've seen replication fail a couple of times in the last two months, with an error message like,

Apply Error 1590: The incident LOST_EVENTS occured on the master. Message: error writing to the binary log

Looking closer into the logs on the replica, I noticed that it was trying to execute an event (we have a heartbeat event that is ENABLED on the primary, and should be in a SLAVESIDE_DISABLED state on the replica(s). Looking closer, I realized the failing replica has this event in ENABLED state, and since the server is running as ``--read-only` it fails while trying to run the event. There are errors like below in the replica logs.

2016-02-28 20:55:00 16040 [ERROR] Event Scheduler: [database_name@%][database_name.update_heartbeat_event] The MySQL server is running with the --read-only option so it cannot execute this statement

I tried dropping the event on the primary, but it had no effect on the replica, presumably because replication is in failing at this point.

Any pointers to help fix this (besides creating a new readonly replica from the primary?)

Thanks

1

Looks like AWS RDS has a bug where when creating a read-only replica of a mysql database with events, it creates the events on the replica with ENABLED status instead of SLAVESIDE_DISABLED.

I verified this by recreating the failing replica. When it created the replica from the snapshot, it put in the events with the wrong state.

Dropping and recreating the event(s) on the primary resolved the issue by creating the same events with the correct status.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.