The FreeBSD Diary

Yesterday I finished installing 8 Raptor drives into the Dual Opteron box and set up a RAID-10 array with two hot spares. The process began on Tuesday night when I found I could not sleep. What else does one do when sleepless? Install drives into cages.

I crept down into the basement, and removed the 6 empty drive cages from the sleeping Dual Opteron (now with a hostname of opti). I took only 6 of the 8 cages because two of them contained drives from the initial setup and testing of the server. The first photo below shows the drive cages and the screws I obtained from Computer Recyclers, a local store. Each drive cage has 6 screws and I made sure they were all used.

Also shown below is a close up of one of the 8 Raptor drives. The funding for these drives was obtained from the readers of this websites and FreshPorts.

The third image is of the drives partially inserted into the server. Yes, that's 6 raptor drives. Why only six showing? The top left drive slot contains the system drive from my previous testing. I wanted to keep that until I knew that the 3Ware card was finding the drives correctly. The bottom right drive cage is stuck. When I press the button, it does not release the lever. Hopefully, I can fix this before I deploy the server.

As previously mentioned, 3Ware donated a 9550SX-8LP. This is a half length /low profile SATA II RAID Controller with 8 ports. The first step is to replace the metal bit at the end of the card. The one supplied with the card will not fit into a 2U case. 3Ware do provide a smaller version and it is easily replaced, the only tool required being a screwdriver.

The next step is to install the BBU (Battery Backup Unit). This add-on allows the controller to use write-caching for optimal performance. Should system power fail, the battery preserves the contents of the controller cache memory. When power is restored, the cached data is written to disk. In short, you should be using a BBU.

Installing the BBU is pretty simple. 3Ware provide very good instructions in the installation guide, (3ware9550SXInstallGuide.pdf) available from their website, for the card. The battery clips onto the card, using three existing snap-points. The memory module attaches to a connector and is held in place using a a small plastic screw. Then you plug the battery into the memory module and you're ready to go. The photo at right shows the BBU mounted on the card. The battery is to the left. Immediately to the right of that is the memory module. The plastic screw is at the top right of that unit.

In the photo below, the battery is on the right, and you can see how the plastic post holds the memory module to the card.

The cable attachment was the most time consuming part of the setup process. I took the time to label each cable so that they could be easily identified should we need to replace one. Each SATA cable is labeled (one of zero to seven: 0..7) and attached to the controller port of the same number. The other end of the cable is attached to the SATA connector on the back plane. When a SATA drive cage is inserted into the case, it connects to the back plane, where it gets both power and data. The drives in the case are labeled 4..7 across the top, and 0..3 across the bottom. The photograph to the left shows the cables as then enter from the front of the case (on your left) and attach to the SATA controller (top right). Also visible in this photograph are the two red SATA cables (top left) that run to the two drives used during the initial testing. Those cables, and the drives, have since been removed.

I have some cable routing issues. These cables (as shown in the photograph to the left) are fairly long. The excess length is laid between the drive fans and the drives. That may impede the air flow. I'll make sure to minimize any issues. It may be better to place the extra cables elsewhere in the case.

When I first booted the system with all eight drives, drive 1 was not found by the 3Ware card. I was pretty sure this was a cable issue. I had tried moving the cables into a less air-resistance location, and when I did, I dislodged the cable for drive 1. I detached it from the back plane and then plugged it back in. Then all was well.

During the boot process, I pressed ALT-3 when the 3Ware notice appeared. Then I added the first 6 drives into a single cluster, and named it PrimaryCluster. The other two drives were left as hot spares. I designated the cluster as RAID10. When this process was finished, I noticed this message, which seemed rather odd at the time.

I placed a FreeBSD 6.1-RELEASE CD in the CD-ROM drive and continued with the booting process. The installer started up. I ran through the install, and rebooted. Up came FreeBSD on the RAID10 cluster! For what it's worth, the elapsed time between booting from CD and then booting up into 6.1-RELEASE was 11 minutes. Contrary to what you can read on the 3Ware website and in their documentation, FreeBSD installs onto the card without downloading any special drivers. need to upgrade firmware. ensure NCQ is enabled. test drive activity and drive failure lights.

The following is an extract from /var/run/dmesg.boot:

3ware device driver for 9000 series storage controllers, version: 3.60.02.012
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x303f mem 
           0xfc000000-0xfdffffff,0xfa020000-0xfa020fff irq 26 at device 2.0 on pci2
twa0: [FAST]
twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue:
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-8LP, 8 ports, 
		   Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024

The RAID controller presents a single drive to the operating system. Note that the manufacturer of da0 is AMCC, which bought 3Ware.

Aug 10 15:01:11 opti kernel: da0 at twa0 bus 0 target 2 lun 0
Aug 10 15:01:11 opti kernel: da0: <AMCC 9550SX-8LP DISK 3.01> Fixed Direct Access SCSI-3 device
Aug 10 15:01:11 opti kernel: da0: 100.000MB/s transfers
Aug 10 15:01:11 opti kernel: da0: 200241MB (410093568 512 byte sectors: 255H 63S/T 25527C)

You may find the following interesting as well:

$ mount
/dev/da0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/da0s1e on /tmp (ufs, local, soft-updates)
/dev/da0s1f on /usr (ufs, local, soft-updates)
/dev/da0s1d on /var (ufs, local, soft-updates)
$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/da0s1a    496M     38M    418M     8%    /
devfs          1.0K    1.0K      0B   100%    /dev
/dev/da0s1e    496M     12K    456M     0%    /tmp
/dev/da0s1f    176G    120M    162G     0%    /usr
/dev/da0s1d    8.6G    236K    7.9G     0%    /var
$

• timing tests
• upgrade to 6.1-STABLE
• upgrade the 3Ware firmware
• more timing tests
• ensure NCQ is enabled
• more timing tests
• test drive activity and drive failure lights on the front panel

Oh, and I have to find a colo for the box....