Mysql error : Slave SQL Could not execute ...on table

Question

Im running a percona cluster of 3 nodes.

Yesterday I needed to throw some more memory in them, so I had to power down all nodes Now I booted up and then my 3rd Node, Gave me this error :

140701 23:40:51 [ERROR] Slave SQL: Could not execute Update_rows event on table database_prod.cron_schedule; Can't find record in 'cron_schedule', Error_code: 1032; handler error HA_ERR_KEY_NOT_FOUND; the event's master log FIRST, end_log_pos 206, Error_code: 1032
140701 23:40:51 [Warning] WSREP: RBR event 2 Update_rows apply warning: 120, 69137050

Whats my options regarding fixing this error. And what does it actually mean.

Answer 1

The 3rd node can not update record in cron_schedule because the record is missing. That could happen because some binlog events were lost when the servers crashed.

First try to skip those errors and start replication. Run this on the faulty slave:

pt-slave-restart

The script skips binlog events that return error and start replication. In some cases a slave is heavily inconsistent and replication never recovers. In that case you need to rebuild the slave. But if replication is restarted and running proceed further.

Now you need to find all inconsistencies and sync data on the master and its slaves.

First run pt-table-checksum on the master(add user/password options if needed):

pt-table-checksum -q

It will show you which tables are inconsistent i.e. have different data on the master and slaves.

Then let pt-table-sync fix the inconsistencies. Run this on the master:

pt-table-sync --execute --replicate percona.checksums --noforeign-key-checks localhost

Afterwards you may want to run pt-table-checksum again to make sure the slaves are in-sync