4

I have a pretty big PostgreSQL 9.1.3 running on Ubuntu 10.04. The data is spread over multiple tablespaces = physical drives.

One of these drives is gone, so the directory of that tablespace doesn't exist any more. e.g.: I have lost the directory where the symlink in "pg_tblspc/176967555" linked to.

Well. Status: After a restart, that DBMS comes up with no error. Accessing that particular database is not possible though

psql: FATAL: could not open file pg_tblspc/176967555/

I tried to simply create these folders as empty, but then the PG wants a PG_VERSION and pg_filenode.map file in that directory, which I can not simply create.

90% of the data in the affected database is stored in other, sane tablespaces. But I can't access any table in the database, because some of the tables were stored on the now-vanished tablespace.

My goal is to read data from the unaffected tablespaces. It would be fine if postgres would just drop anything that lied on that tablespace.

I recovered most files from the lost tablespace directoy (e.g. pg_tblspc/176967555/). When I put that recovered folder back in place, PG is still compaining about a missing file when accessing that DB - one file which I could not recover.

Could starting the DBMS with zero_damaged_pages=true help to just ignore the missing files? If zero_damaged_pages intended to be used for the ''missing file'' szenario? edit: No luck - it will still complain about the missing file:

set zero_damaged_pages = true;
SET
postgres=# \connect problemdb ;
FATAL:  could not open file "pg_tblspc/176967555/PG_9.1_201105231/123304298/135285149": No such file or directory

What are my options?

Should continue I to try to recover the database with the broken tablespace? This discussion seems to provide some tips on how to restore when single files are missing. Can I somehow create these files with dd ?

Should I try to get the needed tables from the binary files with pg_filedump ?

Someone disussed the option to import a tablespace to a new database on the postgres mailinglist in 2009, but there seems to be no way.

Isn't this a pretty normal crash scenario: some file has vanished - and you still wan't to access the tables stored in the other files?

Many thanks in advance for any help. Steve

  • 1
    Your best (probably only) option is to restore your backups (which is standard the answer to a "pretty normal crash scenario") – a_horse_with_no_name May 6 '12 at 9:42
  • 1
    I cringed when I read the words "multiple tablespaces = physical drives". Please consider RAID for future deployments. We have hundreds of servers, which aren't always replaced on the targeted four-year replacement schedule, so we must have a drive failure at least once per month. Since we use RAID with a hot spare and hot replacement, this rarely affects performance, much less causing a need to go to our backups. – kgrittn May 6 '12 at 13:30
  • 2
    Thanks. Two of the tablespaces are RAIDs. I used one tablespace for unimportant tables on a single SSD.. because I thought that that drive could die and the important stuff is on the save RADI-backed tablespaces. I now learned the hard way, that loosing one tablespace can result in no access the whole database. :-/ – alfonx May 6 '12 at 18:22
3

The very first thing to do is to make a copy of whatever data files you still have, and to keep it and any backups safe until long after your recovery effort is complete. Please read this (short) Wiki page:

http://wiki.postgresql.org/wiki/Corruption

Once you have done that, you can attempt various recovery strategies without fear that you will be worse off for the attempt, beyond the time required to try it. In general I recommend carefully following one of the techniques described in the documentation -- attempts to cut corners or to be creative often lead to corruption. Only a seasoned expert with a good understanding of PostgreSQL internals should attempt to deviate from the documented steps.

You didn't describe your backup strategy; details of what is available there may suggest alternatives you would not otherwise have.

Ultimately, if you have data of value which is not backed up, you may need to hand-edit the system tables to eliminate references to lost tablespace. This is not for the faint of heart. There are a number of companies with which you can contract for such services, many of whom have experience with recovery from catastrophic hardware failure like this.

http://www.postgresql.org/support/professional_support/

I am not affiliated with any of these companies.

  • 2
    Thanks. Most helpful has been this (commandprompt.com/blogs/alvaro_herrera/2011/03/…) blog entry by one of these companies. And yes.. it wasn't for the "faint of heart". ;-) And still no success. – alfonx May 6 '12 at 18:25
  • "you can attempt various recovery strategies" Could you describe some of these strategies? – Cerin May 24 '17 at 2:45
1

After making an image of the drive that my data was stored on, I found an easy solution to my problem: the entire pg_tblspc folder was gone.

My steps from there was as follows:

  • Create a new empty pg_tblspc folder.
  • When I did that, I ran into another error: another folder was missing. So I created that folder.
  • I tried starting the service again, and a third folder was missing. So I created that folder.

And, that seems to have worked!

  • Thanks for sharing. By "And, that seems to have worked!" you mean that you were able to start PG again? Which version where you using? – alfonx Jan 23 '18 at 8:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.