The gear we’re testing today:

Dell Equallogic PS6500 (7200rpm disks ~48 drives)
EMC VNXe3200 (7200rpm disks ~24 drives)
Dell/EMC SC4020 (ssd version ~12 drives)
45Drives S45 (7200rpm ~45 drives)
45Drives Q30 (ssd version ~12 drives)
Supermicro 45 bay JBOD (7200rpm disks ~45 drives)
My local machines ssd (just for fun ~1 drive)
All backed by a 10gig fiber network

For this test we’ll be primarily using CrystalDiskMark, but later we do end up using some old school linux dd commands. All CrystalDiskMark readings were done over iSCSI connections to backend storage devices, and had defaults set to: 8gb test file, 32 queues, 8 threads, 5 passes

…One other important note, crystaldiskmark appears to have a bug when testing the 45drives Q30, it was flagging about 300mb/s read speeds even while pegging out a 10gig pipe. This leads me to believe there is an unknown bug in the software, and as such I’ve normalized that result in the bar graphs by just setting it to its theoretical max.

The results!

First up we have all networked devices (iSCSI 10gig link). Note that the SC4020, and both 45drive units capped out the max write speeds on the 10gig link, with only the 45drive S45 unit not maxing out the 10gig link on read speeds.

Next up we have the same chart, but it now includes my local disk ssd, as well as a direct attached (SAS) jbod device. Notice how the JBOD device wins across the board, only because of its lack of hitting the 10gb link cap speed.

An interesting note, most likely to the benefit of ZFS, is that random read and write performance across all devices. Here we can see a few things much more clearly, as well as something closer to real world use. Notice how the traditional spinning disk devices score very low, while ssds score fairly well. Another interesting thing is how well ZFS handled randoms across the spinning disk device, actually outperforming both purely SSD arrays, most likely because of the higher spindle count.

I also did some non networked raw benchmarking between the 45drives units using the linux dd command to test ssd vs spinning disk (12 vs 45 drives). Results showed a 17% performance increase when using ssd drives, at 3.1GB/s and 2.2GB/s respectively. These results are based on a single sequential write.

One final note on CPU usage and the 45Drive units. The 45Drive units nearly maxed out CPU usage on their respective units while using ZFS for raidz2 and compression lz4 only. The S45 unit has two processors at 8 cores each and would peak at around 70% cpu usage while the Q30 with only one processor would hit 80% usage. Usage spikes always matched random write speed tests. As with anything ZFS related, memory was in full use for ARC cache.