I originally posted this at http://blogs.sun.com/brendan/entry/my_sun_storage_7410_perf.
As part of my role in Fishworks, I push systems to their limits to investigate and solve bottlenecks. Limits can be useful to consider as a possible upper bound of performance, as it shows what the target can do. Finding and properly analyzing these limits is a useful exercise, as I can determine the kernel code or hardware that is the bottleneck, leading to product changes and better overall performance. I thought these results would make for some interesting blog posts if I could explain the setup, describe what the tests were, and include screenshots of these results in action. (Update: I've included results from colleagues who have tested in the same manner.)
To summarize the performance limits that I found for a single Sun Storage 7410 head node (our top performing ZFS storage appliance):
| Workload | 7410 (Barcelona) | 7410 (Istanbul) |
|---|---|---|
| NFS streaming read from DRAM | ~1.90 Gbytes/sec | ~3.06 Gbytes/sec |
| NFS streaming read from disk | 1.04 Gbytes/sec | 2.03 Gbytes/sec |
| NFS streaming write to disk | 563 Mbytes/sec | 932 Mbytes/sec |
| NFS 1b read IOPS from DRAM | 281,000 IOPS | 506,000 IOPS |
| NFS 4k read IOPS from DRAM | 191,000 IOPS | 404,000 IOPS |
| NFS 512b reads from L2ARC | 128 - 260 us | |
| NFS 512b sync writes to SLOG | 137 - 181 us | |
| NFS IB/RDMA streaming read from DRAM | N/A | 2.93 Gbytes/sec |
| NFS IB/RDMA streaming read from disk | N/A | 2.11 Gbytes/sec |
| NFS IB/RDMA streaming write to disk | N/A | 984 Mbytes/sec |
| CIFS streaming read from DRAM | ~1.03 Gbytes/sec | |
| CIFS streaming read from disk | 849 Mbytes/sec | |
| CIFS streaming write to disk | 620 Mbytes/sec | |
| CIFS 1b read IOPS from DRAM | 203,000 IOPS | |
All tests are performed on Ethernet (usually 10 GbE) unless otherwise specified ("IB" == InfiniBand).
Like many products, the 7410 will undergo software and hardware updates over time. This page currently has results for:
- 7410 Barcelona: The initial release, with four quad-core AMD Opteron CPUs (Barcelona), and the 2008.Q4 software.
- 7410 Istanbul: The latest release, with four six-core AMD Opteron CPUs (Istanbul), and the 2009.Q3 software.
I should make clear that these are provided as possible upper bounds. These aren't what to expect for any given workload, unless your workload was similar to what I used for these tests. Click on the results to see details of the workloads used.
These are also the limits that were found with a given farm of clients and JBODs. It's possible the 7410 could go faster with more clients and more JBODs.
Updated 3-Mar-2009: added CIFS results.
Updated 22-Sep-2009: added column for 7410 Istanbul. Results will be added as they are collected.
Updated 12-Nov-2009: added Cindi's InfiniBand results.