Jumbo frames performance with 10GbE iSCSI

This is yet another voice to add to the opinion that jumbo frames are worth the effort.

This is especially true in a 10GbE iSCSI environment doing a hardware refresh of hosts, storage and switches or a pure greenfield deployment.

The ease of configuring jumbo frames across your environment is helped by a couple of things:

Most 10GbE switches ship with an MTU of 9214 or 9216 out of the box, so are ready for host and storage to be tuned to match. The Arista 7150 I have tested are configured out of the box this way.

Changing vSphere virtual switches and VMkernel interfaces to a large MTU size is now very easy via the vSphere client and Web Client.

On with the testing.

Hardware I tested with:

vSphere host: Cisco UCSC-C220-M3S, 16-core, 196GB Memory, Intel X520 Dual Port 10Gb SFP+ Adapter. This was running ESXi 5.1 Update 1.

Switch: Arista 7150S-24 (24 SFP+ ports)

Storage: Nimble Storage CS260G (10GbE iSCSI)

This was not meant to be an all encompassing test across all known workloads. Just some synthetic sequential benchmarks to see what difference existed between a default MTU of 1500 bytes vs 9000 bytes for iSCSI.

I set up a VM with Windows Server 2012, 4 x vCPU, 8GB RAM, 100GB system disk and a 100GB data disk (for iometer). This was on a 1TB iSCSI VMFS5 datastore presented from the Nimble array.

Using the good old IOMeter, I setup a basic sequential read test of a 2GB file with 32 outstanding I/O’s and then ran 4K, 8K and 32K tests for 5 minutes (once with ESXi vSwitches/VMkernel interfaces and Nimble ‘s data interfaces at 1500 MTU and then once with them set to 9000 MTU). The Arista was set to an MTU of 9214 across all ports at all times (the default)

jumbo frames

The results showed a consistent 10%+ performance increase in both throughput and bandwidth.

It would be difficult not to recommend jumbo frames as a default configuration for new 10GbE iSCSI or NAS environments.

For more great in-depth reading on jumbo frames and vSphere, make sure to check out Michael Wesbter’s (@vcdxnz001) posts on his blog:




Optimising vSphere path selection for Nimble Storage

THIS POST IS NOW OBSOLETE. Use Nimble Connection Manager instead

This post is mostly for documentation or a quick how-to for changing the default path selection policy (PSP) so it is more suitable for Nimble arrays.

For Nimble Storage by default, ESXi 5.1 uses the ALUA SATP with a PSP of most recently used (MRU)

Nimble recommends that this be altered to round robin.

From the CLI of each ESXi host (or from the vMA):

esxcli storage nmp satp set -s VMW_SATP_ALUA -P VMW_PSP_RR

After this point any volume presented from the Nimble array and configured as a VMFS datastore will appear with round robin load balancing and I/O active on all paths.


If you had previously added volumes before changing the default PSP, you can change all existing Nimble volumes to round robin with PowerCLI

#Get all hosts and store in a variable I have called $esxihosts

 $esxihosts = get-vmhost

#Gather all SCSI luns presented to hosts where the vendor string in the LUN description equals “Nimble” and set the multi-pathing policy

Get-ScsiLun -VmHost $esxihosts | where {$_.Vendor -eq "Nimble"} | Set-ScsiLun -MultipathPolicy "roundrobin"

Nimble also make a recommendation around altering the IOps per path setting for round robin. This by default is 1000 IOps before changing data paths.

The official recommendation it to set it to zero (0) for each volume on each host.

Nimble support confirms that this ignores IOps per path and bases path selection on queue depth instead.

So from esxcli on each host, do the following which will search for the string Nimble on all storage devices presented to the host and then loop through and set IOps to zero

for i in `esxcli storage nmp device list | grep Nimble` ; do esxcli storage nmp psp roundrobin deviceconfig set --type "iops" --iops=0 --device=$i; done

I haven’t tested or implemented this as of yet.

I believe Nimbles internal support community has a single powercli script to cover everything here.

It would not be much work to take the per host esxcli stuff into powercli with get-esxcli and aggregate it all together.

Cleaning up “Already used” desktops in VMware View

The “Already used” error is a common problem that exists with using floating (non-persistent) desktops in a completely stateless way (i.e you have assigned the pool desktops to refresh on log-off)

However, a user can (in some circumstances) shutdown or reset their floating desktop and this does not give the View Management server a chance to refresh the desktop back to its “known good” internal disk ready for another user.

This leaves the desktop in an error state, which required manual admin intervention or scheduled scripts to handle the issue.


Of course you should be using AD group policy, you can then remove the ability of the user to shutdown the PCoIP windows desktop leaving them with just the ability to log-off.

But in reality I have still seen the “Already used” error crop up due to administrators haphazardly and incorrectly resetting floating desktops and power or individual server issues causing pools on local disk to come up in this state.

With View 5.1.2, VMware have added a an LDAP attribute called pae-DirtyVMPolicy

This is a per pool setting that allows the administrator to control how “Already used” desktops are treated.

There are three policy settings:

pae-DirtyVMPolicy=0 – This is the default behavior of leaving the desktop in the error state and not available for use.

pae-DirtyVMPolicy=1 – This allows desktops that were not cleanly logged off to be available without being refreshed. The desktop is available in the pool for another user.

pae-DirtyVMPolicy=2 – This setting will automatically refresh a desktop in the “already used” state and make it available again in the pool.

To change the setting on a pool, fire up ADSI Edit on a View Connection server and connect to dc=vdi, dc=vmware, dc=int (see screenshot)


Browse down to Server Groups and you will see your desktop pools on the right pane


Double click a pool name to edit the attributes


Find the LDAP attribute pae-DirtyVMPolicy in the list and change its value from <not set> to 1 or 2 (depending on your requirement)

Hopefully this can help ease some of the administrative burden of large floating desktop pools and the small issues they can have.

For more information on VMware View 5.1.2, see the release notes here:


Change the IP address of VMware vCloud Director 5.1 virtual appliance

This blog post is mostly for my own lab documentation, but hopefully someone else may find it useful.

The vCloud Director appliance is built upon 64-bit SuSE Enteprise linux with an embedded version of Oracle 11g Express, which means its setup and troubleshooting are a little different from the supported Redhat Enterprise and Oracle platform.

If you had set up your vCD appliance initially with DHCP on both primary and console proxy interfaces, you may run into issues changing it later, as there are few places to look.

The first thing is to SSH into the vCD appliance and logon as root (you will have to enable root logon=yes in /etc/ssh/sshd_config and restart the sshd daemon first)

Stop the vCloud services; service vmware-vcd stop

cd to $VCLOUD_HOME/etc (which should be /opt/vmware/vcloud-director/etc)

Use vi or vim to edit the global.properties file and change the following IP addresses to what you need


Out of the box the vCD appliance does not have the correct Oracle variables set.

So the following commands are necessary:

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe



You should then be able to successfully run sqlplus with the default username and password for the vCloud DB of vcloud/VCloud.

sqlplus vcloud/VCloud

At the SQL> prompt type:

select name from cells;

This will tell you the name of the director cell you are on

Then type: (substituting the appropriate IP address and cell name details)

update cells set primary_ip=’your-primary-ip’ where name=’name-of-directorcell’;

It should say 1 row updated

Exit from sqlplus

Restart the vCloud services; service vmware-vcd start

You should now be able to access the vCloud admin site on the new primary IP

Thinapp Factory – fixing adding Workpool error

VMware just released a new fling, a Thinapp workflow automation tool called Thinapp Factory.

You can read all about it here and download it:http://labs.vmware.com/flings/thinapp-factory

When setting up the Thinapp Factory virtual appliance, pay close attention to the details it is asking you for during the OVF setup.

If you do not supply the correct details, when creating a workpool (new clean VM’s to build thinapps on), you may get a 404 error (page not found) or a HTTP Error 500 Internal server error. (see two images below)



The quick solution is to power down the Thinapp factory appliance and edit its settings. Go to the options tab and vApp properties.

In here make sure you have correct the vCenter server specified (IP or resolvable host name). Also make sure the account you are connecting with is a vSphere admin on that vCenter server.


Scroll down and correctly fill out the datacenter and cluster details. These are exactly the same as your datacenter and cluster names in your vSphere environment (see next image)

You can leave the Resource Pool area blank unless you want the Workpool VM to go somewhere else other than the root resource pool. You will also to provide a datastore name for these VM’s to live.


I have seen other tips and docs stating you should use IP/hostnames for these sections, but they do not work in my environment.

After you have correctly modified these for your environment, power up the appliance and log in via the web interface.

Go to settings and then Workpool.

You should then be able to add a new VM, either letting Thinapp Factory build from an ISO or use an existing VM from your vSphere environment.

And hopefully from there, no more problems getting you first automated Thinapp packages built.

As always, it’s always useful to read the install docs for any other info:

Teradici PCoIP firmware 3.5 USB performance comparsion

Recently, Teradici released version 3.5 firmware for PCoIP Zero clients based on the Portal processor.

One of the main enhancements was the ability to fully utilize the USB 2.0 capable hardware that always existed in these devices, but was restricted to USB 1.1 performance by the previous firmware.

Version 3.5 firmware removes this restriction when using a PCoIP Zero client against a VMware View desktop.

I thought I would do a quick test to see what the real world differences were between version 3.4 and 3.5 firmware on a Wyse P20 Zero client.

The Test setup
Wyse P20 Zero client (on gigabit network)
VMware View 5.0 and vSphere 5.0 Infrastructure
Windows XP Virtual Desktop (1 vCPU and 1.5Gb RAM) running in a View 5.0 floating pool
Teradici PCoIP Management Console 1.7

I took a basic 2GB Sandisk USB memory stick and connected it to the Wyse P20 running V3.4 firmware. I then copied a 150MB test file from the local disk of my VM to the memory stick.

I then logged off the desktop and refreshed it so as not to influence the performance via caching etc.

The Wyse P20 was then updated to V3.5 firmware, pushing it out from the Teradici Management console.

Logging back into a fresh View XP desktop, I copied the same 150MB file to the USB memory stick.

The results:

V3.4 firmware: 4m:35s to copy 150MB file

V3.5 firmware: 1m:48 to copy 150MB file

So, almost a four fold increase in USB performance

This will make the Teradici based zero clients more acceptable in situations requiring moving files via USB memory stick and removable hard disks, as well as other applications involving USB based CD/DVD burners.

In my environment, this is the best feature of the latest V3.5 firmware – a must have.

Configuring iSCSI with ESXCLI in vSphere 5.0

This is mostly me learning my way around some of the new namespaces in esxcli as part of vSphere 5.0

If you want to know what’s new in esxcli in vSphere 5.0, please read these two posts from Duncan Epping and William Lam.

I wanted to see what needed to be done to configure a load balanced iSCSI connection with two VMkernel portgroups.

So here goes; All done against just a standard vSwitch.

Enable software iSCSI on the ESXi host
~ # esxcli iscsi software set --enabled=true

Add a portgroup to my standard vswitch for iSCSI #1
~ # esxcli network vswitch standard portgroup add -p iSCSI-1 -v vSwitch0

Now add a vmkernel nic (vmk1) to my portgroup
~ # esxcli network ip interface add -i vmk1 -p iSCSI-1

Repeat for iSCSI #2
~ # esxcli network vswitch standard portgroup add -p iSCSI-2 -v vSwitch0
~ # esxcli network ip interface add -i vmk2 -p iSCSI-2

Set the VLAN for both my iSCSI VMkernel port groups - in my case VLAN 140
~ # esxcli network vswitch standard portgroup set -p iSCSI-1 -v 140
~ # esxcli network vswitch standard portgroup set -p iSCSI-2 -v 140

Set the static IP addresses on both VMkernel NICs as part of the iSCSI network 
~ # esxcli network ip interface ipv4 set -i vmk1 -I -N -t static
~ # esxcli network ip interface ipv4 set -i vmk2 -I -N -t static

Set manual override fail-over policy so each iSCSI VMkernel portgroup had one active physical vmnic
~ # esxcli network vswitch standard portgroup policy failover set -p iSCSI-1 -a vmnic0
~ # esxcli network vswitch standard portgroup policy failover set -p iSCSI-2 -a vmnic3

Bond each of the VMkernel NICs to the software iSCSI HBA
~ # esxcli iscsi networkportal add -A vmhba33 -n vmk1
~ # esxcli iscsi networkportal add -A vmhba33 -n vmk2

Add the IP address of your iSCSI array or SAN as a dynamic discovery sendtarget
~ # esxcli iscsi adapter discovery sendtarget add -A vmhba33 -a

Re-scan your software iSCSI hba to discover volumes and VMFS datastores
~ # esxcli storage core adapter rescan --adapter vmhba33