Recovering from catastrophes

Last night, my main file server had major issues, its running Solaris10, as far as I can tell, the damage was caused from a rouge paste, of course I didn’t figure it out till this morning when I put the pieces together after I had started to reinstall the OS, I think rm’d /usr/libc.*. Okay well the box wouldn’t boot, so I booted the install cdrom, and backed up some key files that make the next step possible.

This server has two SVM raid file systems, one raid 5 the other is raid 0, of course most of the data is not backed up, but most of it isn’t important, just more work to re-download than I wanted to deal with. Okay everyone I have asked about this in the past say its impossible to recover from or it’s a lot of work. Well I figured an easy way to do it if you have the necessary config files. So here are the steps to recovering them easy.

#1 before you start, calm down, relax you should be able to recover most of the data on the system, just time is what you really lost.

#2: boot the install disk

ok boot cdrom –s

Now mount the root filesystem at point /a usually this is /dev/dsk/c0t0d0s0

mount /dev/dsk/c0t0d0s0 /a

Now find that we have access to the old files on the system, and your attempts to recover the system have failed, lets grab some files to make our lives easier after we re-install

cd /etc/
tar cvf /etc.tar *
cd /kernel
tar cvf /kernel.tar *

grab a copy of /export/home you may have to mount it, but I always feel a little better knowing I have a copy of it.

HINT: Don’t forget any web pages you have on /var/www or /var/apache*

tar up anything else you need off the box while you have it sitting there. It would not be a bad idea to keep a copy of all those config files and burn them on a CD for emergencies like this.

Find a nice ftp server somewhere to hold your files while you reinstall and move the files to there temporary home. Preferably not on a file system or drive attached to the box that is going be installed.

After you have retrieved and made a copy of all the files you want and the config files, you can now reinstall

HINT: choose custom install or it will wipe your fist hard disk before you realize it.

And choose to retain the data on your /export/home file system, even though you have a copy of it, its better to play it safe.

After the install is finished, login in as root. Now we have to replace some files so that the system will see the metadevices.

First download the tar files you made earlier, specifically etc.tar and kernel.tar

Then make a directory and untar them.

cd /
mkdir /recover
cd /recover
tar xf /etc.tar
tar xv /kernel.tar

we have our files so we can access them first lets put back SVM files in /kernel

# cd kernel/drv
# cp md.conf /kernel/drv
# cd ../..

now the etc files

# cd etc/lvm
# cp * /etc/lvm

the files are back in place, cross your fingers, and reboot the box. Hopefully in 3 minutes or so the box will be back up waiting for you to login

Lets see what we have now. (just listing one device but you get the idea)

# metastat d1
d1: Concat/Stripe
Size: 33456128 blocks (15 GB)
Stripe 0: (interlace: 256 blocks)
Device Start Block Dbase Reloc
c0t3d0s0 0 No Yes
c0t4d0s0 0 No Yes
c0t8d0s0 0 No Yes
c0t9d0s0 2160 No Yes

Device Relocation Information:
Device Reloc Device ID
c0t3d0 Yes id1,sd@SSEAGATE_ST34371W_SUN4.2GJDM274270D61LR
c0t4d0 Yes id1,sd@SSEAGATE_ST34371W_SUN4.2GJDM950230C920R
c0t8d0 Yes id1,sd@SSEAGATE_ST34371W_SUN4.2GJDL27464056DDZ
c0t9d0 Yes id1,sd@SSEAGATE_ST34371W_SUN4.2GJDU320350210TG
#

Okay well drives now appear to be back, lets fsck them to be sure.

# fsck /dev/md/dsk/d1
** /dev/md/dsk/d1
** Last Mounted on /zones
** Phase 1 - Check Blocks and Sizes
** Phase 2 - Check Pathnames
** Phase 3 - Check Connectivity
** Phase 4 - Check Reference Counts
** Phase 5 - Check Cyl groups
314259 files, 13093131 used, 3364863 free (316399 frags, 381058 blocks, 1.9% fragmentation)

Now all that is left is to copy the right lines from /recover/etc/vfstab to /etc/vfsab so auto mounting once again occurs.

Well crisis averted, your data is safe, go eat, have some coffee, or perhaps go sleep, and don’t forget to send me $1 for every gigabyte of data this how-to has saved.

For more blogs related to OpenSolaris and Solaris.