Vinum Logical Volume Manager Performance Measurement

First Round of Testing

This round of testing was done under FreeBSD 3.2-RELEASE on a Dell PowerEdge 1300 with one Pentium III processor running at 450Mhz, and with 1MB L2 cache. The machine also had 1GB ECC RAM, and the four 10kRPM IBM UltraStar 9LZX drives were connected to the server via the internal Adaptec AIC-7890 controller (equivalent to the AHA-3940UW) or the DPT SmartRAID IV PCI caching RAID controller (with 64MB cache).

Note that only striping was tested. There was not time to test mirroring, mirroring+striping, or RAID-5.

Conclusion: vinum wins, hands-down. The DPT SmartRAID IV, even in pure striping mode, just can't keep up with the much faster CPU. It's not yet clear how close to optimal the DPT SmartRAID IV implementation is as compared to vinum, but for now it looks like vinum wins.

Second Round of Testing

This round of testing was conducted under FreeBSD 4.1-STABLE on a Dell PowerEdge 1300 with one Pentium III processor running at 450Mhz, and with 1MB L2 cache. The machine also had 1GB ECC RAM, and five Quantum Viking 2 9GB hard drives. The only controller used was a DPT SmartRAID V Century with 16MB of on-board ECC RAM cache, either using the on-board RAID capabilities or as a pass-through controller (the machine would not boot when the drives were connected to the on-board Adaptec controller). We were fortunate to be able to test on virtually the same hardware platform (albeit with a different version of the OS, so that we could get support for the DPT controller), so as to make the numbers as comparable as possible.

We have only tested RAID-5, and striping (RAID-0), figuring that these are the extremes of reliability versus performance, and that other choices such as RAID-1 (a.k.a. "mirroring") and RAID-1+0 (a.k.a. "mirroring+striping") would probably fall somewhere between. We have summarized these latest results and compared them against the previous tests, and have drawn some conclusions.

Note that the "natural" RAID-5 stripe size for the DPT controller is 32KB, but these tests were run with a stripe size of 128KB, and produced much, much better results.

Conclusions

Speed-wise, vinum wins, hands-down. There is just absolutely no comparison whatsoever. Neither the DPT SmartRAID IV nor the SmartRAID V can BEGIN to keep up with the much, much faster CPU.
However, I still don't trust RAID-5 under vinum (it has had a long and colorful history of surprisingly negative interactions with software that it should not, such as "softupdates"), and I have not yet had a chance to test vinum under failure mode conditions (where at least one disk of a RAID set has failed). In cases where I care about reliability and where I want or need to use RAID-5, I'll probably be forced to choose an alternative option that almost certainly will not perform as well.
The first chart of the second test shows the importance of high spindle speeds in random read environments, although it also shows that when you have a lot of simultaneous processes accessing the disk subsystem, you can partially compensate for lack of speed for individual disks by simply adding more of the slower disks. This chart also clearly shows that you want to have as much RAM cache in your dedicated controller as possible, since the SmartRAID IV outperformed the SmartRAID V.
Moral: More spindles are good
Moral: Fast spindles are better
Moral: More RAM cache is better
The second chart of the second test shows that a higher number of slower spindles can outperform a smaller number of faster spindles, especially when you get into large numbers of simultaneous processes, since you can have more processes running simultaneously without tripping over the toes of other processes. The SmartRAID V just plain sucks without enough RAM cache.
Moral: Fast spindles are good
Moral: More spindles are better
Moral: More RAM cache is better
It is with the third and fourth charts of the second test that we really start to see the real benefit of having more spindles. With enough spindles going, vinum just FLIES through these tests. Even the SmartRAID V seriously benefits from having more spindles, and handily beats the SmartRAID IV in writes, even though it is desperately short of cache RAM.
Moral: More spindles are better

However, this is not the whole story. We also need to look at tests that come a little closer to simulating real-world data access patterns than rawio, which simply runs four separate tests that are each 100% dedicated to either sequential or random I/O and reading or writing.

Real-world Application: Mail Servers

Application Mix
In my experience, mail servers tend to have a random I/O pattern, but it tends to be split almost exactly 50/50 reads versus writes (a mail message comes in and is written the the users mailbox, then the message is read and it almost certainly never gets touched again, with the possible exception of being deleted). Keep in mind that your absolute number one killer is I/O latency, especially with regards to synchronous meta-data updates.
If you look at the random read and random write charts in the second test, and you visually average those results, you'll notice that vinum with the five 7200RPM Quantum Viking II drives is notably faster than vinum with the four 10kRPM IBM 9LZX drives. The implication is that having more spindles tends to be more important than having faster spindles, although ideally you should have both more spindles and they should all be as fast as is feasible.
However, if you have to economize, the place to spend the money is on getting more spindles, even if they have to be a little slower. Of course, all the spindles should be identical (at least in performance characteristics, if not size), otherwise the whole system is likely to be yoked down to the performance characteristics of the least powerful drive, and you will have wasted any money you spent on buying anything better.
Of course, if you have an existing system which needs to be upgraded, think about adding more spindles first. It's usually relatively easy to add more spindles to a machine, but faster spindles may not always be available.
Spindles versus controllers
Take care not to add too many spindles to a single channel on a single controller -- for absolute minimum latency, a good rule of thumb is four HDAs (Hard Disk Assemblies, or what I've been calling "spindles") per controller channel, although I usually find that for the kinds of applications I see, five or six HDAs per channel per controller is not too excessive.
If you have external RAID devices with their own built-in controller(s), this limitation needs to be applied internally to the device as well as with respect to the total number of devices and how many channels on how many host adaptors that they are connected to.
Of course, all of this assumes SCSI or FibreChannel disk devices, which are known to perform reasonably well under heavy multi-user/multi-processing load (i.e., hundreds of processes would ideally like to be simultaneously reading and writing user mailboxes). I won't even bother to attempt to test IDE/ATAPI drives in this kind of application mix, since we already know that they perform very poorly for these kinds of loads. Perhaps there are RAID devices which communicate to the host via SCSI or FibreChannel but which use IDE/ATAPI drives internally, and which can be made to perform reasonably well in this kind of application mix and at a decent price, but I have not yet personally seen them.
Stripe Size
The average mail message size and the average mailbox size will have a HUGE impact on what would be the best overall choice of stripe size, which you ideally want to be at least as large as the average I/O operation (but probably not more than 256KB, where experience shows that you tend to start getting diminishing returns). The result is that each access can (on average) be completed by a single head in a single operation, so that you maximize the total number of simultaneous different operations you can have ongoing at any one point in time, and you minimize overall average latency.