1

Thanks to the recent weather/floods, we lost many of our servers. This included our central database server as well as the server that holds our backups. The database server was running Oracle 12c enterprise on RHEL6. We offload our backups to an offsite facility once a month. Long story short, our most usable backup is over 3 weeks old. The database server was sent to a data recovery place, who managed to get us partial files, some of which we believe contain our datafiles. But these files are incomplete and lack any / all header information. Is there any way of salvaging any data out of these files at all? Opening up the files with a hex editor displays the data we need, but we are taking about a 50 GB database and mining through the datafiles manually would take forever. We have pretty much given up recovering anything but I thought I would just ask the question here in case someone else has had a similar experience. I am aware of Oracle DUL tool (already opened an SR with Oracle in this regard and this was their advice) but the fees they charge are exorbitant. We are small company - we have already paid a tidy sum for the data recovery service. We simply cannot afford to pay what Oracle are asking for, as much as we would like to. Can anyone suggest a possible alternative please? And we are currently reviewing our data backup policies but as I mentioned - we are a small business with limited resources - and we didn't plan for natural disasters like we should have - we are going to be more careful in the future though.

UPDATE : Right turns out the hard drives themselves are fine though the LVMs are corrupt. I managed to rebuild the partitions with the help of testdisk. But I can't seem to mount the drives. I get an error "NTFS Signature Missing". Looking at dmesg it looks like mount has looked for etx3 and ext4 super blocks and didn't find them. So I am assuming the partitions are corrupt.

Thanks in advance.

Alex

Improve this question
6
  • You were running 12c in production? Already? Madness. Anyway, oracleodu.com/en is probably your best bet if they've updated for 12c. Other than that, pay Oracle.
    – Philᵀᴹ
    Dec 30, 2013 at 0:01
  • @FreshGrinchOfSO Different client. I work as a consultant. I inherited this DB and it's fair share of issues. I am not a DBA. Just a programmer.
    – Alex
    Dec 30, 2013 at 3:22
  • Definitely a cautionary tale about appropriate off-site backup policies and procedures.
    – BBlake
    Dec 30, 2013 at 13:31
  • @Alex well, I feel for you. It's not going to be a cheap problem to solve
    – Philᵀᴹ
    Dec 30, 2013 at 13:40
  • @FreshGrinchOfSO Cheers mate. Hopefully we will get something sorted out.
    – Alex
    Dec 30, 2013 at 17:02

2 Answers 2

Reset to default
2

Right,

For anyone else who might come across this post - I am just adding this information for the sake of completeness.

It took me a long time to get this sorted. I booted off of a SystemRescueCD and ran testdisk. It managed to recover my LVM partitions, which I couldn't mount due to damage to the disks. I tried numerous file recovery tools. R-Studio was probably the most helpful, it managed to recover file fragments. I also ran Scalpel version 2.0. This version allows for specification of minimum file size, which was important in my case since I knew how big the files were. Scalpel took roughly 3 days, but we managed to recover our 5 data files. However without any of the other files (controlfiles, archived redo logs etc) recovery was still a challenge. I managed to create new controlfiles and rebuild the spfile from a previous copy of the pfile. I ended up starting the database with resetlogs. So no far we haven't had any reports of missing data.

We were incredibly lucky. While I would say that even in the worst imaginable scenario there might be hope for recovery, it is in no way an excuse to eschew prescribed backup methods. I cannot overstate how lucky we were that we managed to salvage our data, after having to deal with a malfunctioning RAID controller, corrupt partitions and missing files. My employers have now taken a more serious and pragmatic view towards offsite backups. but I wouldn't want to see anyone in this position. Like they say - prevention is better than cure.

For reference : Oracle Data File Headers in Hex : 00 A2 00 00 FF C0 ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? 7A 7B 7C 7D Oracle Data File Footer : 00 00 00 01 The question marks represent characters which will be different for every data file. Scalpel allows you to specify wildcards. R-Studio also allows defining a specific file type and searching for that file type, but I didn't try it.

Thanks to everyone who took time to share their advice. I am grateful to you all.

Regards

Alex

Improve this answer
0

For recovery you might want to check out ORA-600: http://www.ora600.be Good luck.

Improve this answer
1
  • 1
    Doesn't support 12c though
    – user1822
    Dec 31, 2013 at 11:47

Not the answer you're looking for? Browse other questions tagged or ask your own question.