The FreeBSD Diary
Providing practical examples since 1998If you buy from Amazon USA, please support us by using this link.
Converting a system to RAID 7 February 2007
This may well be the last article I write on this server. My existing development machine is still on 4.11. Today I will start the migration process to another server, which is already running 6.2-STABLE. In the process, I will add a RAID card to the server, and transfer the existing OS from a SATA drive to the RAID-1 cluster (based on two 160GB IDE drives).
Note that although I am using a RAID array here, it appears to the OS as a single HDD, so this process could equally apply to moving to a new bigger HDD. For your information, I will be using a 3Ware 7006-2 from 3Ware.
The main reason for putting RAID into this machine is that it will become my main development machine and take over the duties of two machines that now run FreeBSD 4.11 (which has recently hit End-of-Life and no more security updates will occur for 4.x).
We now join the migration, already in progress.
Is the RAID card there?
As of this point, I've installed the two new IDE drives and hooked them up to the 3Ware card. fter boot, I see this in the output from dmesg:
twe0: [GIANT-LOCKED] twe0: 2 ports, Firmware FE7X 1.05.00.063, BIOS BE7X 1.08.00.048
That is my RAID card, just waiting to be set up. Good. Now I will reboot and go through the RAID BIOS and set up the cluster.
By the way, I've also installed sysutils/tw_cli and sysutils/3dm, two 3Ware utilities. I have mentioned them in a previous article on the 3Ware 9550SX card.
Preparing the system for the transfers
We will be using FreeSBIE to do the data transfer between the old and new HDD. Why? It is easier to copy an HDD when it is unused and with no chance of the data being modified. In short, the system is offline and copying the HDD is guaranteed to be correct.
From there, I used the instructions for Swapping boot drives around, written way back in August 1999. All I needed was the details for tar'ing up one file system and untar'ing it on the other. For this clean and easy technique, I will boot from something other than the SATA drive and the two IDE RAID drives. I could have used an HDD which already had FreeBSD on it. I choose instead to use a FreeSBIE live CD. FreeSBIE 2.0 gives me FreeBSD 6.2 and all the tools I need to massage the data according to my will. First, I boot up the system from the CD. Then I su to root, and move to my home directory. Then I create two directories:
mkdir new old
These two directories will form the starting point for my transfers. I will mount the existing SATA drive under the old directory and the new RAID-1 system under the new directory.
I need mount points for the various slices. So I did this:
cd old mkdir root var tmp usr cd .. cd new mkdir root var tmp usr
As you will see in the problems section later on, I forgot to do this, but you won't:
chmod 1777 ~/old/tmp
I started sysinstall and then used it to configure the new HDD (twed0). I click on the following links once in sysinstall:
I will now label the new HDD. Again, from the FreeBSD Configuration Menu, I click on Label. Ensure the right disk is selected, twed0 in my case.
Now I will create several slices and assign them some mount points. sysinstall will actually mount the slices at the mount points specified, and which we created earlier in this process. The short hand here is:
C = Create | 1 GB | FS | /mnt/root C = Create | 2 GB | swap C = Create | 2 GB | FS | /mnt/tmp C = Create | 2 GB | FS | /mnt/var C = Create | All (142GB) | FS | /mnt/usr W = Write Yes
Here is what the screen looked like after issuing the Write command.
FreeBSD Disklabel Editor Disk: twed0 Partition name: twed0s1 Free: 0 blocks (0MB) Part Mount Size Newfs Part Mount Size Newfs ---- ----- ---- ----- ---- ----- ---- ----- twed0s1a /mnt/root 1024MB UFS2+S Y twed0s1b swap 2048MB SWAP twed0s1e /mnt/tmp 2048MB UFS2+S Y twed0s1f /mnt/var 2048MB UFS2+S Y twed0s1g /tmp/usr 142GB UFS2+S Y
And in case it helps you, here are the the mounted filesystems:
[dan@ngaio:/mnt] $ mount /dev/ad6s1a on / (ufs, local) devfs on /dev (devfs, local) /dev/ad6s1e on /tmp (ufs, local, soft-updates) /dev/ad6s1f on /usr (ufs, local, soft-updates) /dev/ad6s1d on /var (ufs, local, soft-updates) fdescfs on /dev/fd (fdescfs) polo:/usr/ports/distfiles on /usr/ports/distfiles (nfs) /dev/twed0s1a on /mnt/root (ufs, local, soft-updates) /dev/twed0s1e on /mnt/tmp (ufs, local, soft-updates) /dev/twed0s1f on /mnt/var (ufs, local, soft-updates) /dev/twed0s1g on /tmp/usr (ufs, local, soft-updates) [dan@ngaio:/mnt] $
I will not be copying over devfs or fdescfs. NFS mounts should also not be copied over. In short, look only for the mount points associated with the HDD in question. In this case, ad6.
Of note were some interesting directories I had never seen before:
[dan@ngaio:/mnt] $ find . . ./tmp ./tmp/.snap ./var ./var/.snap ./root ./root/.snap ./usr [dan@ngaio:/mnt] $
Here is how I mounted the various slices of the SATA drive:
mount /dev/ad6s1a ~/old/root mount /dev/ad6s1e ~/old/tmp mount /dev/ad6s1f ~/old/var mount /dev/ad6s1d ~/old/usr
Similarly, for the new RAID-1 cluster:
mount /dev/twed0s1d ~/new/root mount /dev/twed0s1e ~/new/tmp mount /dev/twed0s1f ~/new/var mount /dev/twed0s1g ~/new/usr
Everything is ready. Let's transfer the data. I will do this one file system at a time. Let's start with the root partition:
tar --one-file-system -c -f - -C ~/old/root/ . | tar xpvf - -C ~/new/root/
The tar on the left will do a cd to ~/old/root/ and start tar'ing everything up, staying within the one file system. The -c is to create. The -f designates the output file, this case STDOUT.
The tar on the right will take extract, retain permissions, and display each file name as it is extracted. The -f - indicates STDIN as the source of the tarball. tar will do a cd to ~/new/root before it starts this process.
This process needs to be repeated for each mount point:
tar --one-file-system -c -f - -C ~/old/var/ . | tar xpvf - -C ~/new/var/ tar --one-file-system -c -f - -C ~/old/tmp/ . | tar xpvf - -C ~/new/tmp/ tar --one-file-system -c -f - -C ~/old/usr/ . | tar xpvf - -C ~/new/usr/
That sounds just too easy. Well, there is one gotcha. If anything refers to the old HDD by device, then it will break. You must change the new /etc/fstab before you reboot from the new RAID array. In my case, the file I need to edit is ~/new/root/etc/fstab and the changes required reflect what happened when I was using the Disk Label Editor. That is, I made sure that /dev/twed0s1a was mounted at /, twed0s1b was swap, twed0s1e was mounted at /tmp, etc.
Then I shutdown the system with a shutdown -p now and disconnected the SATA drive. During the power up process, I removed the FreeSBIE CD from the CD drive and let the system boot from the RAID card. Apart from the minor permission problem mentioned below, it went flawlessly.
The /tmp permission problem
The system would not boot from the RAID cluster the first time I tried. The problem was I had root on /dev/twed0s1d instead of /dev/twed0s1a. I went into FreeSBIE and rebooted. I went into the Disk Label Editor, deleted the slice, and created a new one, this time with s1a, not s1d. I had to redo the tar | tar process, but that didn't take too long. I also had to edit ~/new/root/etc/fstab to point at twed0 instead of ad6.
The next reboot worked, but there was a problem with /tmp permissions. A chmod 1777 fixed that. The next reboot showed no problems.