By default in ESXi the Round Robin IOPS limit is set to 1000 meaning that the path only changes after 1000 I/O operations have occurred. The result is basically the same performance as running a single path in my experience. Adjusting the limit from 1000 to 1 can result in much better performance.

The first step is to enable SSH on ESXi. Click Manage in the top left of the WebGUI and go to the Services tab. Select TSM-SSH and click Start. Now you can SSH in to ESXi using PuTTY or a bash terminal.

You can check the current IOPS setting and the naa ID we’ll need with this command

esxcli storage nmp device list

In the output you should find a line similar to this one. I’ve made the important information Blue:

naa.6589cfc000000f0033f8cf3c39a68aa7
Device Display Name: FreeNAS iSCSI Disk (naa.6589cfc000000f0033f8cf3c39a68aa7)
Storage Array Type: VMW_SATP_ALUA
Storage Array Type Device Config: {implicit_support=on; explicit_support=off; explicit_allow=on; a lua_followover=on; action_OnRetryErrors=off; {TPG_id=1,TPG_state=AO}}
Path Selection Policy: VMW_PSP_RR
Path Selection Policy Device Config: {policy=iops,iops=1000,bytes=10485760,useANO=0; lastPathIndex=2: NumIOsPending=0,numBytesPending=0}
Path Selection Policy Device Custom Config:
Working Paths: vmhba66:C10:T0:L0, vmhba66:C0:T0:L0, vmhba66:C15:T0:L0, vmhba66:C5:T0:L0
Is USB: false

To change the IOPS from 1000 to 1 enter this command, replacing XXXX with the naa ID:

for i in `esxcfg-scsidevs -c |awk '{print $1}' | grep naa.XXXX`; do esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=$i; done

Run the previous command again to confirm the IOPS have changed from 1000 to 1 and you’re done.

Benchmarks

Here is a comparison of the performance difference in a Windows Server 2008 R2 VM with a paravirtualized storage adaptor. I’m running 4 gigabit Ethernet paths however my performance is limited by my 3 old WD Green drives in RaidZ1.

1000 IOPS

1 IOPS

FreeNAS Graphs: