i know this question was asked before, but i am at a point where i am really stuck and do not know how to continue with this problem and i would really like to know, what causes this error to come up.
we had a replication which worked without any problem. our setup includes one database master and four slaves, all running percona mysql, same version (5.5.x). the other day our database master died because of a hardware issue. so we went on and made a slave the new master. should not be much of a problem and was indeed no problem before. but now after getting things up again, i keep getting lot's of log entries in my syslog like:
Mar 15 13:17:25 dbslave3 mysqld: 120315 13:17:25 InnoDB: Error: page 263337 log sequence number 34159788445 Mar 15 13:17:25 dbslave3 mysqld: InnoDB: is in the future! Current system log sequence number 1096541424.
Ok, so i used xtrabackup -- which i used for this things before -- to create a dump from the new master and used the dump for applying it on the slaves. without success. still the same problem with this error message. next i tried the following:
i took my site offline and made sure, that all data was replicated to all slaves -- indeed: replication is working! no data get's lost, every slave has all the data it should have.
next i did a
reset slave on the slaves and on the master i did a
reset master. in my opinion this should reset any binlogs and start over again with fresh sequence numbers. next i did a
show master status and used the new sequence file and number for the
change master to ... command on the slaves. i activated replication again and took my site online again.
without success! still the same problem. so my biggest question:
WTF is going on!? how can this error occur on:
- a fresh set-up slave using xtrabackup
- after a binlog reset on master and slave
how to solve it? i mean: i read a lot about inserting gigabytes of data in the database, that "the sequence pointer can catch up" ... but this sounds a lot like a dirty hack and can't be a real solution to this problem, no!? and my biggest concern: is the problem indeed solved using this dirty hack?
any other ideas what i could do? dumping the data using mysqldump and importing it again is no option btw., because the database is so big, that it would take to long to import it again.
what i also did is: dumping one of the slaves, including: deinstalling mysql completly, removing all mysql related stuff including binlogs, databases, etc. and setting it up again -- without success. that's the fresh install i am asking about above.