Take the 2-minute tour ×
Database Administrators Stack Exchange is a question and answer site for database professionals who wish to improve their database skills and learn from others in the community. It's 100% free, no registration required.

I have a pretty big PostgreSQL 9.1.3 running on Ubuntu 10.04. The data is spread over multiple tablespaces = physical drives.

One of these drives is gone, so the directory of that tablespace doesn't exist no more. e.g.: I have lost the directory where the symlinc in "pg_tblspc/176967555" linked to.

Well. Status: After a restart, that DBMS comes up with no error. Accessing that particular database is not possible though

psql: FATAL: could not open file pg_tblspc/176967555/

I tried to simply create these folders as empty, but then the PG wants a PG_VERSION and pg_filenode.map file in that direcoty, which I can not simply create.

90% of the data in the affected database is stored in other, sane tablespaces. But I can't access any table in the database, because some of the tables were stored on the now-vanished tablespace.

My goal is to read data from the unaffected tablespaces. It would be fine if postgres would just drop anything that lied on that tablespace.

I recovered most files from the lost tablespace directoy (e.g. pg_tblspc/176967555/). When I put that recovered folder back in place, PG is still compaining about a missing file when accessing that DB - one file which I could not recover.

Could starting the DBMS with zero_damaged_pages=true help to just ignore the missing files? If zero_damaged_pages intended to be used for the ''missing file'' szenario? edit: No luck - it will still complain about the missing file:

set zero_damaged_pages = true;
SET
postgres=# \connect problemdb ;
FATAL:  could not open file "pg_tblspc/176967555/PG_9.1_201105231/123304298/135285149": No such file or directory

What are my options?

Should continue I to try to recover the database with the broken tablespace? This discussion seems to provide some tips on how to restore when single files are missing. Can I somehow create these files with dd ?

Should I try to get the needed tables from the binary files with pg_filedump ?

Someone disussed the option to import a tablespace to a new database on the postgres mailinglist in 2009, but there seems to be no way.

Isn't this a pretty normal crash scenario: some file has vanished - and you still wan't to access the tables stored in the other files?

Many thanks in advance for any help. Steve

share|improve this question
1  
Your best (probably only) option is to restore your backups (which is standard the answer to a "pretty normal crash scenario") –  a_horse_with_no_name May 6 '12 at 9:42
1  
I cringed when I read the words "multiple tablespaces = physical drives". Please consider RAID for future deployments. We have hundreds of servers, which aren't always replaced on the targeted four-year replacement schedule, so we must have a drive failure at least once per month. Since we use RAID with a hot spare and hot replacement, this rarely affects performance, much less causing a need to go to our backups. –  kgrittn May 6 '12 at 13:30
1  
Thanks. Two of the tablespaces are RAIDs. I used one tablespace for unimportant tables on a single SSD.. because I thought that that drive could die and the important stuff is on the save RADI-backed tablespaces. I now learned the hard way, that loosing one tablespace can result in no access the whole database. :-/ –  alfonx May 6 '12 at 18:22
add comment

2 Answers

The very first thing to do is to make a copy of whatever data files you still have, and to keep it and any backups safe until long after your recovery effort is complete. Please read this (short) Wiki page:

http://wiki.postgresql.org/wiki/Corruption

Once you have done that, you can attempt various recovery strategies without fear that you will be worse off for the attempt, beyond the time required to try it.

You didn't describe your backup strategy; details of what is available there may suggest alternatives you would not otherwise have.

Ultimately, if you have data of value which is not backed up, you may need to hand-edit the system tables to eliminate references to lost tablespace. This is not for the faint of heart. There are a number of companies with which you can contract for such services, many of whom have experience with recovery from catastrophic hardware failure like this.

http://www.postgresql.org/support/professional_support/

I am not affiliated with any of these companies.

share|improve this answer
1  
Thanks. Most helpful has been this (commandprompt.com/blogs/alvaro_herrera/2011/03/…) blog entry by one of these companies. And yes.. it wasn't for the "faint of heart". ;-) And still no success. –  alfonx May 6 '12 at 18:25
add comment

I suppose you have no backups... Adventurous one you are.

Copy everything to another computer before doing anything.

Maybe you could recreate the same tablespace structure using for example a virtual machine and then copy the directory back to broken one. It may work, but I did not try this at home.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.