0

I was doing migrations while the server was still accessing the slave DB. I have a master / slave setup. After I restarted the server, I realized I couldn't connect to the SLAVE anymore. I went into the SLAVE and tried restarting Postgres, this is what it returned:

$ sudo /etc/init.d/postgresql start
 * Starting PostgreSQL 9.3 database server                                       
 * The PostgreSQL server failed to start. Please check the log output:
2014-12-16 22:14:20 UTC LOG:  database system was interrupted while in recovery at log time 2014-12-16 20:57:20 UTC
2014-12-16 22:14:20 UTC HINT:  If this has occurred more than once some data might be corrupted and you might need to choose an earlier recovery target.
2014-12-16 22:14:20 UTC LOG:  entering standby mode
2014-12-16 22:14:20 UTC LOG:  redo starts at 0/38620168
2014-12-16 22:14:20 UTC LOG:  consistent recovery state reached at 0/38680BF0
2014-12-16 22:14:20 UTC LOG:  database system is ready to accept read only connections
2014-12-16 22:14:20 UTC PANIC:  heap2_redo: unknown op code 32
2014-12-16 22:14:20 UTC CONTEXT:  xlog redo UNKNOWN
2014-12-16 22:14:21 UTC LOG:  startup process (PID 18357) was terminated by signal 6: Aborted
2014-12-16 22:14:21 UTC LOG:  terminating any other active server processes

Any idea on what could be causing this and what I can do to fix this? I'm going into a PANIC frenzy.

  • What exact versions are you upgrade from and to? Which are you upgrading first? – jjanes Feb 18 '15 at 20:04
  • @jjanes Thanks for the quick reply, I'm not upgrading to anything. I already had streaming replication set up. I made some changes to the tables and I couldn't connect to the slave. The master is still working but the slave refuses to start – deadlock Feb 18 '15 at 20:09
3
+25

http://www.postgresql.org/docs/9.3/static/release-9-3-3.html

Fixing this required changing the WAL record format for tuple freezing. While this is no issue for standalone servers, when using replication it means that standby servers must be upgraded to 9.3.3 or later before their masters are. An older standby will be unable to interpret freeze records generated by a newer master, and will fail with a PANIC message. (In such a case, upgrading the standby should be sufficient to let it resume execution.)

| improve this answer | |
  • both the master and the slave are running 9.3.2, I didn't update the version of postgres, I simply made a bunch of table migrations, added new columns and tables. – deadlock Feb 18 '15 at 20:17
  • Are you positive? This is exactly the error message you would get when replaying WAL from 9.3.3 into a 9.3.2 server. – jjanes Feb 18 '15 at 21:19
  • Yeap, I ran psql -V on both the master and the slave and both machines returned the following: psql (PostgreSQL) 9.3.2 – deadlock Feb 18 '15 at 21:24
  • To clarify, I had my web servers running while I was doing the table migrations. The changes propagated to the master no problem, but perhaps during the transmission to the slave, a READ operation on the slave was made and therefore corrupted the DB? What do you think? – deadlock Feb 18 '15 at 21:25
  • 2
    psql -V might be misleading, as it tells you what client you are using and not necessarily what server. They could be different. start psql connecting to your server, and then do select version();. This will tell out about the server. In any case, you shouldn't be using 9.3.2, it had some nasty (but thankfully mostly rare) bugs. – jjanes Feb 19 '15 at 2:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.