2

I have streaming replication setup on Postgresql 9.1 with two machines on AWS EC2. The master has a cron job that does a daily base backup using the wal-e tool and uploading it to Amazon's S3 storage. The WAL files are also archived to S3 by using the same tool.

At some point, when I had to replace the slave instance, I ran a wal-e based script that loads the base backup from S3 and writes it to $PGDATA. After that when I started up the database, I got the following errors:

... Apr 10 10:15:29 dbc postgres[25865]: [12-1] 2014-04-10 10:15:29 UTC LOG: restored log file "00000005000000B800000089" from archive Apr 10 10:15:31 dbc postgres[25865]: [13-1] 2014-04-10 10:15:31 UTC LOG: restored log file "00000005000000B80000008A" from archive Apr 10 10:15:33 dbc postgres[25865]: [14-1] 2014-04-10 10:15:33 UTC LOG: restored log file "00000005000000B80000008B" from archive Apr 10 10:15:47 dbc postgres[25865]: [15-1] 2014-04-10 10:15:47 UTC LOG: restored log file "00000005000000B80000008C" from archive Apr 10 10:15:47 dbc postgres[25865]: [16-1] 2014-04-10 10:15:47 UTC FATAL: could not access status of transaction 0 Apr 10 10:15:47 dbc postgres[25865]: [16-2] 2014-04-10 10:15:47 UTC DETAIL: Could not open file "pg_subtrans/2214": No such file or directory. Apr 10 10:15:47 dbc postgres[25865]: [16-3] 2014-04-10 10:15:47 UTC CONTEXT: xlog redo hot_update: rel 1663/24577/5279107; tid 22/32; new 22/84 Apr 10 10:15:47 dbc postgres[25863]: [1-1] 2014-04-10 10:15:47 UTC LOG: startup process (PID 25865) exited with exit code 1 Apr 10 10:15:47 dbc postgres[25863]: [2-1] 2014-04-10 10:15:47 UTC LOG: terminating any other active server processes

The server does not start up.

Does this mean that my base backup is corrupted? Or does it mean that some WAL files are corrupted?

The master server is running Postgres 9.1.12 (Debian Wheezy), but the newly launched slave is running Postgres 9.1.13 (Debian Wheezy) - this happened because when installing the new machine, only the minor upgraded version of Postgres 9.1 was available in the repositories. My understanding is that this should not be a cause of the problem.

Interestingly, if I do an rsync of the master's data directory to the slave, and then start the slave with streaming replication setup, the slave comes up fine.

Because of this it is not clear if I have a corruption in my DB data directory (which is on an AWS EBS drive) or if there is something wrong with the backup tool wal-e. Any help is appreciated.

  • It certainly won't be a difference of minor versions. What's the script you're using? Where's the base-backup you're applying WAL to taken from, and how was it created? – Craig Ringer Apr 10 '14 at 13:37
  • The base backup is taken from an S3 bucket by the wal-e tool. It was created by the same tool running on the master. – donatello Apr 10 '14 at 14:10
  • If you restore the base backup without doing further WAL replay, does it start up OK? – Craig Ringer Apr 11 '14 at 1:05
1

I've experienced this problem with wal-e up to 0.9.2. Manually creating the pg_subtrans directory seems to have got replication working again.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.