While not perfect (it does not fork, and therefore is completely single-threaded, and it does not require that all meta-data be written synchronously), it does seem to be at least a plausible first-pass simulation of an Internet mail or USENET news system, in that it creates lots of relatively small files (between one-half and 10KB in size), does operations on them, and then deletes them in rapid succession. Note that it there is now an official "port" for postmark in FreeBSD 4.x-CURRENT, under /usr/ports/benchmarks/postmark/.
The goal here is to get multiple people to submit data to be included in these tables, and I have not personally performed all the testing shown here. The company or personal name of the author is enclosed in square brackets at the end of the description of each system configuration that was tested, and for the original results from Network Appliance, I used the tag "[NetApp]".
Their original white paper is at http://www.netapp.com/tech_library/3022.html, but I'll replicate the test results here (to make the comparisons easier): PostMark was configured in four different ways:
The NFS benchmark was performed on a Sun Microsystems Ultra 1/170 with 256 Mbytes RAM running under the Solaris 2.5 operating system. The following configurations were tested:
|Transactions per second||36||2000||63||23||139||253||94||271||1851||54||458||1724||29||107||2000||82||485||492||980||72||149|
|Data read (Kbytes/sec)||115.67||4880||199.73||74.13||441.71||799.91||296.10||850.70||6092.8||171.77||1495.04||5488.64||93.04||336.36||6328.32||268.32||1566.72||332.02||3174.40||235.40||489.66|
|Data written (Kbytes/sec)||118.27||7330||204.22||75.79||451.64||817.89||302.75||869.82||6236.16||175.64||1525.76||5611.52||95.14||343.92||6471.68||280.35||1638.40||346.92||3317.76||245.96||511.62|
|Transactions per second||15||438||29||14||76||176||20||62||98||35||142||228||16||16||416||29||349||36||666||15||23|
|Data read (Kbytes/sec)||29.93||663.64||56.60||27.05||177.68||383.41||49.24||142.98||237.29||67.30||318.81||504.14||32.34||37.40||1126.4||82.00||737.57||99.14||1495.04||38.95||66.57|
|Data written (Kbytes/sec)||54.22||1530||102.54||49.00||321.88||694.58||89.20||259.02||429.87||121.91||577.55||913.29||58.58||67.75||2048||154.75||1392.64||187.09||2816.00||73.51||125.63|
|Transactions per second||335||30||14||74||169||26||56||83||35||139||228||16||19||564||31||310||33||613||15||22|
|Data read (Kbytes/sec)||613.03||73.19||35.05||204.72||446.69||73.52||153.28||232.66||85.86||329.79||606.47||41.01||49.86||1679.36||90.20||819.89||94.42||1628.16||42.68||69.72|
|Data written (Kbytes/sec)||1160||101.17||48.46||282.98||617.45||101.62||211.88||321.60||118.69||524.98||838.31||56.69||68.92||2324.48||129.65||1177.60||135.72||2334.72||61.35||100.21|
|Transactions per second||57||25||86||1333||18||15||564||24||277||28||574||14||13|
|Data read (Kbytes/sec)||151.17||60.92||237.10||3471.36||44.79||39.13||1669.12||65.43||708.74||77.21||1484.80||38.81||40.19|
|Data written (Kbytes/sec)||208.96||84.21||327.75||4802.56||61.92||54.09||2314.24||93.79||1015.93||110.68||2129.92||55.63||57.61|
The thing that impresses me most about these tests is that the first one shows primarily cache effects under FreeBSD with softupdates and Linux with async writes, but overall the ancient (and slow) Pentium Pro beat the Sun UltraSPARC (with twice as much RAM) in every test! The Linux machine in the first test also shows major cache effects with the DPT SmartRAID V controller (which continues to have a somewhat dimished effect throughout the tests), and its performance drops quite precipitously with the larger tests, where FreeBSD with softupdates tends to do measurably better (Linux had the advantage of being on a single CPU and avoiding SMP, FreeBSD had the advantage of being totally unloaded).
Unfortunately, the Sun Ultra 5 tests with UFS Logging were set up in the most pathological configuration possible -- the metadata log device was mirrored to two different stripes on the same disk, and both of which were on the same disk as the data device. Yet, except for the surprising results with subdirectories, UFS Logging still outperformed UFS at least a little, and for small filesysets outperformed it by quite a lot (by a factor of approximately 3.6). This demonstrates a percentage speed up close to what is visible with FreeBSD and softupdates!
However, the results that NetApp got with their Ultra 1/170 are rather surprising. It makes you wonder just what exactly what the configuration of this machine was, and how they managed to get results that are faster (for the first test) than what I could manage with an Ultra 5, more memory, and a newer version of the OS (with all unnecessary programs disabled). Maybe this was with an extremely fast SCSI disk?
These tests highlight the importance of using a log-structured journaling filesystem, "softupdates" or some other method of safely delaying, grouping, and avoiding writing to disk synchronous metadata operations, in order to improve performance. Note that the Linux-standard practice of mounting volumes "async" is very dangerous, and while it may result in high performance you can get equally high performance with higher reliability using FreeBSD with softupdates.
Interestingly, comparing the respective results of the third and fourth tests, FreeBSD MFS performance appears to be highly dependant on the number of files in a single directory that is being accessed and updated (as one would expect), while Sun Solaris TMPFS performance appears to be dependant solely on the total number and size of files on the filesystem, regardless of how many subdirectories they might be stored in.
Yes, I know that the NetApp F330 and F630 are not the current top-of-the-line models (that would be the F760), but then none of the systems I'm comparing them against are the latest top-of-the-line, either. In fact, most of the systems I'm comparing with have designs that are themselves about as old as the F330 and F630 models, so this is still a reasonably valid comparison. The only thing that's newer about the systems I'm comparing with is the OS (and resulting network libraries, etc...), and NetApp has had just as much opportunity to improve their OS as everyone else has.
Once I'm further along with my vinum testing, I'll also add some postmark testing with it on the PPro 200 machine with softupdates, to see just how far this machine can be pushed without adding any expensive hardware (such as a DPT SmartRAID V controller, although I also hope to test that on this machine).