I originally posted this at http://blogs.sun.com/brendan/entry/unusual_disk_latency.
This screenshot shows two spikes of unusually high disk I/O latency during a streaming write test:
This screenshot is from Analytics on the 7410. The issue is not with the 7410, it's with disk drives in general. The disk latency here is also not suffered by the client applications, as this is ZFS asynchronously flushing write data to disk. Still, it was great to see how easily Analytics could identify this latency, and interesting to see what the cause was.
See this video for the bizarre explanation:
Don't try this yourself...