0

The Setup: An EC2 instance running MySQL 5.6 locally replicating to RDS MySQL. The VPC IP address range is 172.0.0.0/8.

Somehow the slave got out of sync and I'm getting this error when I do SHOW SLAVE STATUS:

Could not execute Update_rows event on table db37708.general; Can't find record in 'general', Error_code: 1032; handler error HA_ERR_END_OF_FILE; the event's master log newprod.000106, end_log_pos 285712203

I compared the master and slave tables and yes, there is some data out of sync. Not schema, just text data. Okay, fine, I'll just run my script that wraps pt-table-checksum and I get this:

[ubuntu@fabersubuntu utils ]$ ./checksumSpecificTable.sh db37708 general
Replica ip-10-3-2-119 is stopped.  Waiting.
Replica ip-10-3-2-119 is stopped.  Waiting.
^C# Caught SIGINT.
            TS ERRORS  DIFFS     ROWS  CHUNKS SKIPPED    TIME TABLE
04-19T12:09:50      0      0        1       1       0  32.282 db37708.general

So I run the script again after adding PTDEBUG=1 to it and I see things like this:

# DSNParser:1638 30974 DBI::db=HASH(0x21aae58) SET @@SQL_QUOTE_SHOW_CREATE = 1/*!40101, @@SQL_MODE='NO_AUTO_VALUE_ON_ZERO,NO_ENGINE_SUBSTITUTION'*/
# DSNParser:1652 30974 DBH info:  DBI::db=HASH(0x21aae58) $VAR1 = {
#   '@@hostname' => 'ip-10-3-2-119',
#   'CONNECTION_ID()' => '351452',
#   'DATABASE()' => undef,
#   'VERSION()' => '5.6.40-log'

My questions are:

  1. Where is this "hostname" coming from? Assuming it's an IP address format, it makes sense that it is waiting because I don't have any hosts in a 10.0.0.0 range.

  2. If I CALL mysql.skip_repl_error, I get another error; apparently I have lots of errors all of a similar type (so far). I assume pt-table-checksum can't do anything for me if replication isn't working, so what would be next step be other than rebuild my RDS slave?

  • Is your firewall on? If that's a reverse resolved IP for (119.2.3.10) (China netblock), it could just be external hacky connection attempts that aren't actually properly logged in – Philᵀᴹ Apr 19 at 17:16
  • Yes, I have a firewall on the EC2 instance as well as a very restricted Security Group. My hypothesis is pt-table-checksum is reading that information from somewhere and trying to connect to it. – faberfedor Apr 22 at 13:50
0

Your Question 1

Hostname is from the output of SHOW PROCESSLIST. From that output it tries to detect the slaves. If this is not the right outcome for you, then you can define the dsns table with the list of hosts and use --recursion-method=dsn

Part two of your question will need more information, but you may be able to help yourself using PTDEBUG=1 pt-table-checksum <params>

You are welcome to update your original question with the output from the debug if you would like help with understanding the output.

Disclosure: I work for Percona

  • I am using --recursion-method=dsn and that hostname is not in the dsns table. – faberfedor Apr 24 at 17:53
  • Is there any particular section of the PTDEBUG=1 output I should post? It's quite lengthy. – faberfedor Apr 24 at 17:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.