Things look quiet here. But I've been doing a lot of blogging at
dan.langille.org because I prefer WordPress now.
Not all my posts there are FreeBSD related.
I am in the midst of migrating The FreeBSD Diary over to WordPress
(and you can read about that here).
Once the migration is completed, I'll move the FreeBSD posts into the
new FreeBSD Diary website.
Yesterday I finished installing 8 Raptor drives into the Dual Opteron box and set up a RAID-10 array with two hot spares.
The process began on Tuesday night when I found I could not sleep. What else does one do when sleepless? Install
drives into cages.
I crept down into the basement, and removed the 6 empty drive cages from the sleeping Dual Opteron (now with a hostname of opti).
I took only 6 of the 8 cages because two of them contained drives from the initial setup and testing of the server.
The first photo below shows the drive cages and the screws I obtained from Computer Recyclers, a local
store. Each drive cage has 6 screws and I made sure they were all used.
Also shown below is a close up of one of the 8 Raptor drives. The funding for these drives was obtained from the readers of
this websites and FreshPorts.
The third image is of the drives partially inserted into the server. Yes, that's 6 raptor drives. Why only six showing?
The top left drive slot contains the system drive from my previous testing. I wanted to keep that until I knew that the
3Ware card was finding the drives correctly. The bottom right drive cage is stuck. When I press the button, it does not
release the lever. Hopefully, I can fix this before I deploy the server.
Setting up the 3Ware 9550SX-8LP
As previously mentioned, 3Ware donated a
This is a half length /low profile SATA II RAID Controller with 8 ports. The first step is to replace the metal bit
at the end of the card. The one supplied with the card will not fit into a 2U case. 3Ware do provide a smaller version
and it is easily replaced, the only tool required being a screwdriver.
The next step is to install the BBU (Battery Backup Unit). This add-on allows the controller to use
write-caching for optimal performance. Should system power fail, the battery preserves the contents
of the controller cache memory. When power is restored, the cached data is written to disk. In short,
you should be using a BBU.
Installing the BBU is pretty simple. 3Ware provide very good instructions in the installation guide, (3ware9550SXInstallGuide.pdf) available from their website,
for the card. The battery clips onto the card, using three
existing snap-points. The memory module attaches to a connector and is held in place using a
a small plastic screw. Then you plug the battery into the memory module and you're ready to go.
The photo at right shows the BBU mounted on the card. The battery is to the left. Immediately to the right of
that is the memory module. The plastic screw is at the top right of that unit.
In the photo below, the battery is on the right, and you can see how the plastic post holds the memory module to the card.
The cable attachment was the most time consuming part of the setup process. I took the time to label each cable so that they could be easily
identified should we need to replace one. Each SATA cable is labeled (one of zero to seven: 0..7) and attached to the controller port of the same
number. The other end of the cable is attached to the SATA connector on the back plane. When a SATA drive cage is inserted into the case,
it connects to the back plane, where it gets both power and data. The drives in the case are labeled 4..7 across the top, and 0..3 across the bottom.
The photograph to the left shows the cables as then enter from the front of the case (on your left) and attach to the SATA controller (top right).
Also visible in this photograph are the two red SATA cables (top left) that run to the two drives used during the initial testing. Those cables, and the
drives, have since been removed.
I have some cable routing issues. These cables (as shown in the photograph to the left) are fairly long. The excess length is laid between the drive fans
and the drives. That may impede the air flow. I'll make sure to minimize any issues. It may be better to place the extra cables elsewhere in the case.
Creating the RAID array
I wanted to mention that all of the above activity occurred in the dining room. Yes, that's a 55 pound 2U server sitting on the table.
That total does not include the eight SATA drives.
My laptop is on the left and I'm using it to read the installation guide. To the right of the monitor is an APC UPS. On top of the
server is the 3Ware card and BBU, just prior to being assembled and inserted. The memory module is sitting on top of a CD. The battery unit
is just above the card.
When I first booted the system with all eight drives, drive 1 was not found by the 3Ware card. I was pretty sure this was a cable issue.
I had tried moving the cables into a less air-resistance location, and when I did, I dislodged the cable for drive 1. I detached it from
the back plane and then plugged it back in. Then all was well.
During the boot process, I pressed ALT-3 when the 3Ware notice appeared. Then I added the first 6 drives into a single cluster,
and named it PrimaryCluster. The other two drives were left as hot spares. I designated the cluster as RAID10. When this process was finished, I noticed this
message, which seemed rather odd at the time.
I placed a FreeBSD 6.1-RELEASE CD in the CD-ROM drive and continued with the booting process. The installer started up. I ran through the install, and rebooted.
Up came FreeBSD on the RAID10 cluster! For what it's worth, the elapsed time between booting from CD and then booting up into 6.1-RELEASE was 11 minutes.
Contrary to what you can read on the 3Ware website and in their documentation, FreeBSD installs onto the card without downloading any special drivers.
need to upgrade firmware. ensure NCQ is enabled.
test drive activity and drive failure lights.
The following is an extract from /var/run/dmesg.boot:
3ware device driver for 9000 series storage controllers, version: 3.60.02.012
twa0: <3ware 9000 series Storage Controller> port 0x3000-0x303f mem
0xfc000000-0xfdffffff,0xfa020000-0xfa020fff irq 26 at device 2.0 on pci2
twa0: INFO: (0x04: 0x0053): Battery capacity test is overdue:
twa0: INFO: (0x15: 0x1300): Controller details:: Model 9550SX-8LP, 8 ports,
Firmware FE9X 3.01.01.028, BIOS BE9X 3.01.00.024
The RAID controller presents a single drive to the operating system. Note that the manufacturer
of da0 is AMCC, which bought 3Ware.
Aug 10 15:01:11 opti kernel: da0 at twa0 bus 0 target 2 lun 0
Aug 10 15:01:11 opti kernel: da0: <AMCC 9550SX-8LP DISK 3.01> Fixed Direct Access SCSI-3 device
Aug 10 15:01:11 opti kernel: da0: 100.000MB/s transfers
Aug 10 15:01:11 opti kernel: da0: 200241MB (410093568 512 byte sectors: 255H 63S/T 25527C)
You may find the following interesting as well:
/dev/da0s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/da0s1e on /tmp (ufs, local, soft-updates)
/dev/da0s1f on /usr (ufs, local, soft-updates)
/dev/da0s1d on /var (ufs, local, soft-updates)
$ df -h
Filesystem Size Used Avail Capacity Mounted on
/dev/da0s1a 496M 38M 418M 8% /
devfs 1.0K 1.0K 0B 100% /dev
/dev/da0s1e 496M 12K 456M 0% /tmp
/dev/da0s1f 176G 120M 162G 0% /usr
/dev/da0s1d 8.6G 236K 7.9G 0% /var
I have a plan. A cunning plan. It includes:
upgrade to 6.1-STABLE
upgrade the 3Ware firmware
more timing tests
ensure NCQ is enabled
more timing tests
test drive activity and drive failure lights on the front panel