The Sunfire X4500 came with 48 500GB drives and 6 disk controllers.  We configured the array into 7 RAIDZ2 sets, then striped the 7 sets together, leaving 4 spares and two root disks.  Since we started with Solaris 5/08, we went with a ufs mirror for the two root disks (Solaris 10/08 lets you use ZFS root pool).  This gives us plenty of redundancy at the disk and controller level.

Next we set up several ZFS filesystems, essentially one filesystem for each website using NFS, and one ZFS volume for each iSCSI share.  For now, we are only using one iSCSI share — this will change soon.

The filesystems exported over NFS have no capacity limits, while the iSCSI volume was thin provisioned at 1TB.  We are currently using only using about 100GB of the iSCSI volume.  In the long run the thin provisioned volume will save us from having to resize the volume and extend the ext3 filesystem that sits on top.

The iSCSI volume was shared to an Redhat RHEL5 box running Open-iSCSI.  We found Open-iSCSI to generally work well until you have a network hiccup.  In our initial tests, a network delay of 3 seconds would cause the SCSI block device to disappear temporarily, freaking ext3 out and remounting read-only.

Our solution to network hiccups was to add a DM-Multipath layer between the iSCSI and ext3 layers on the RHEL box.  Multipath will sense network issues and queue I/O until the issue is resolved, or a configurable timeout expires.  The queuing behavior worked perfectly as we went about unpluging and pluging network cables on the RHEL box.  Side note: Multipath is a much more capable piece of software, allowing a machine to switch from one network path to another if it had redundant network paths to the SAN.

One gotcha to look out for when using a Solaris + RHEL5 + Multipath combo: a bug on the Solaris iSCSI target side causes the SCSI ID to be reported as much longer than Multipath can handle.  We got around the bug by extracting what appeared to be the unique SCSI ID portion.

Performance was near wire-spead for sequential I/O.  Random I/O was fantastic, thanks in large part to the 16GB of RAM the Sunfire was using as a giant disk cache.  The I/O performance was much better than the local SATA that the RHEL box came with.

With the iSCSI share created, shared, all appropriate iptables/ipf rules were set up, and the block device mounted, we then moved all of the OpenVZ containers to this RHEL box.  OpenVZ needed no special configuration — we simply mounted the iSCSI device on /vz using ext3.  All containers have been running smoothly since (except for a bad RAM module, but that was unrelated to iSCSI or OpenVZ).

The NFS shares have also been running smoothly, although we did run into a nasty Solaris ipf issue.  The problem is that after a few days, automountd on RHEL randomly hangs when attempting an additional mount.  Packet captures indicate that ipf is blocking either the inbound or outbound packets for the second TCP connection (we are not using UDP) to the ‘nfs’ port.  Disabling ipf immediately resolves the problem.  We are currently trying to reproduce the issue on a test server.

Even with the few issues that came up, we are still very happy with our new setup.  The Sunfire X4500, ZFS, and Solaris 10 matches many of the essential features of a more expensive “enterprise” solution.