Renaming vSphere Datastore device display names

In vCenter, storage devices or volumes/LUNs present themselves with a unique identifier usually prefixed with eui. or naa.

These are known as Network Address Authority Identifiers and Extended Unique Identifiers. (part of FC and iSCSI naming format standards for LUNs)

However, when looking at them in vSphere and via the esxcli, these unique identifiers are less than useful for identifying which VMFS datastores you are dealing with when looking at paths selection etc.

For that reason, I usually rename each of these via the vSphere client as I add them.

This also benefits when filtering NMP devices from the esxcli

If you have connected to the command line of an ESXi host, you can grep for your renamed display name on a storage device

esxcli storage nmp device list | grep -A 9 VMFSProd

You can see that it picks up the Device Display Name set via the vSphere client as well as returning the 9 rows of available information about that NMP device

Hyper-V converged network configuration with two NICs

Just a script I put together for my Server 2012 Hyper-V hosts that only have two 10Gbe physical ports, where I want to separate various network functions of Hyper-V into separate virtual network adapters.

It also lets me use MPIO with two virtual network adapters dedicated for iSCSI.

This is useful for iSCSI storage vendors who recommend using Windows MPIO with their arrays, e.g HP StoreVirtual and Nimble Storage.

With older versions of Hyper-V (Windows 2008 R2), vendor NIC teaming and iSCSI was not recommended, so you were left with dedicating two physical NICs just for an iSCSI MPIO configuration. This would normally mean you would need at least 4 NICs in your host. (iSCSI and other Hyper-V virtual switch functions)

#Create new NIC team using two existing physical adapters in host, set LB and teaming mode
New-NetLbfoTeam -Name NICTeam1 -TeamMembers pNIC1,pNIC2 -LoadBalancingAlgorithm TransportPorts -TeamingMode SwitchIndependent

#Create new virtual switch using the NIC team created prior. Set to not allow hyper-v mgmt
New-VMSwitch -Name ConvergedHyperSwitch -NetAdapterName NICTeam01 -AllowManagementOS $False -MinimumBandwidthMode Weight

#Create five virtual adapters for various components (LM,mgmt,cluster,iSCSI)
Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "LiveMigration" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "CSV" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "iSCSI01" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "iSCSI02" -SwitchName "ConvergedHyperSwitch"

#If required, set VLAN access for your virtual adapters
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "Management" -Access -VlanId 100
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CSV" -Access -VlanId 100
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LiveMigration" -Access -VlanId 100
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "iSCSI01" -Access -VlanId 140
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "iSCSI02" -Access -VlanId 140

#Set minimum bandwidth in weighting (0-100)
Set-VMNetworkAdapter -ManagementOS -Name "LiveMigration" -MinimumBandwidthWeight 20
Set-VMNetworkAdapter -ManagementOS -Name "CSV" -MinimumBandwidthWeight 10
Set-VMNetworkAdapter -ManagementOS -Name "Management" -MinimumBandwidthWeight 10
Set-VMNetworkAdapter -ManagementOS -Name "iSCSI01" -MinimumBandwidthWeight 20
Set-VMNetworkAdapter -ManagementOS -Name "iSCSI02" -MinimumBandwidthWeight 20

#Set IP addresses/subnet on interfaces 
New-NetIPAddress -InterfaceAlias "vEthernet (Management)" -IPAddress 10.201.201.201 -PrefixLength "16" -DefaultGateway 10.201.1.1 
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Management)" -ServerAddresses 10.200.3.21, 10.200.3.96 
New-NetIPAddress -InterfaceAlias "vEthernet (LiveMigration)" -IPAddress 10.201.201.202 -PrefixLength "16" 
New-NetIPAddress -InterfaceAlias "vEthernet (CSV)" -IPAddress 10.201.201.203 -PrefixLength "16" 
New-NetIPAddress -InterfaceAlias "vEthernet (iSCSI01)" -IPAddress 10.190.201.120 -PrefixLength "24" 
New-NetIPAddress -InterfaceAlias "vEthernet (iSCSI02)" -IPAddress 10.190.201.121 -PrefixLength "24" 

Optimising vSphere path selection for Nimble Storage

This post is mostly for documentation or a quick how-to for changing the default path selection policy (PSP) so it is more suitable for Nimble arrays.

For Nimble Storage by default, ESXi 5.1 uses the ALUA SATP with a PSP of most recently used (MRU)

Nimble recommends that this be altered to round robin.

From the CLI of each ESXi host (or from the vMA):

esxcli storage nmp satp set -s VMW_SATP_ALUA -P VMW_PSP_RR

After this point any volume presented from the Nimble array and configured as a VMFS datastore will appear with round robin load balancing and I/O active on all paths.

If you had previously added volumes before changing the default PSP, you can change all existing Nimble volumes to round robin with PowerCLI

#Get all hosts and store in a variable I have called $esxihosts

 $esxihosts = get-vmhost

#Gather all SCSI luns presented to hosts where the vendor string in the LUN description equals “Nimble” and set the multi-pathing policy

Get-ScsiLun -VmHost $esxihosts | where {$_.Vendor -eq "Nimble"} | Set-ScsiLun -MultipathPolicy "roundrobin"

Nimble also make a recommendation around altering the IOps per path setting for round robin. This by default is 1000 IOps before changing data paths.

The official recommendation it to set it to zero (0) for each volume on each host.

Nimble support confirms that this ignores IOps per path and bases path selection on queue depth instead.

So from esxcli on each host, do the following which will search for the string Nimble on all storage devices presented to the host and then loop through and set IOps to zero

for i in `esxcli storage nmp device list | grep Nimble` ; do esxcli storage nmp psp roundrobin deviceconfig set --type "iops" --iops=0 --device=$i; done

I haven’t tested or implemented this as of yet.

I believe Nimbles internal support community has a single powercli script to cover everything here.

It would not be much work to take the per host esxcli stuff into powercli with get-esxcli and aggregate it all together.

Storage Efficiency with Nimble Compression

In my last post I talked about performance of the Nimble Storage platform I was implementing.

In a contended virtualised environment, storage is often the performance sore point or Achilles heal due to insufficient IOps and poor latency of the storage design.

The other major consideration is how efficient the storage platform is at managing the available capacity and what can be done to “do more with less”

With the Nimble platform I am using – this problem is addressed by using:

• In-line compression on all workloads

Nimble uses a variable block compression algorithm they say can yield a 30-70% saving without altering performance or latency.

• Thin provisioning

Ubiquitous across most array vendors, variations in block size can result in differences in efficiency.

• VAAI UNMAP for space reclaimation

Nimble supports the VAAI primitive UNMAP for block reclaimation. This is done from the esxcli of a vSphere host as per this VMware KB:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

For testing I’ll concentrate on the compression and how it fares for a SQL server in a vSphere 5.1 environment.

I took a test SQL server running Windows 2008/SQL 2008 that had disk utilisation around 88%  (130GB of data on 149GB of available disk)

I then performed a storage vMotion migration from the original VMFS datastore on an HP LeftHand volume to a 200GB thin provisioned VMFS datastore presented from the Nimble array.

The result was a 2.25 x saving in space (compressed from 130GB to 56GB)

Other VMs such as file/print, remote desktop hosts, infrastructure VMs in the test environment have anywhere between 1.7x to 2.0x compression levels.

In addition to the savings on primary volumes, compression is also applied to snapshots.

Who can say no to close to double the effective capacity on all virtualised workloads with no extra datacenter footprint?

Initial Performance Impressions of Nimble Storage

Recently I added a Nimble CS240G to my vSphere production storage environment.

The Nimble is affordable and packs some serious features and performance in a 3u enclosure.

The CS240G has 24TB of raw capacity, comprised of twelve 2TB 7200rpm Western Digital enterprise RE4 SATA drives and four Intel 160GB SSDs (Intel 320)

It has redundant controllers, one active, one on standby (mirrored). There are dual 10GbE ports on each controller as well as dual 1GbE.

You can choose to have the data and management sharing one network or split it out. In my case I went with a redundant management setup on the two 1GbE ports on one VLAN and iSCSI data on the two 10GbE ports on another VLAN.

Nimble does smart things around its ingestion of data. The general process is:

• Inbound writes, cached in NVRAM and mirrored to the standby controller, write acknowledgement is given to the host at this point (latency measured in micro seconds)
• The write IO’s are compressed inline in memory and a 512KB stripe is built (from possibly many thousands of small IO’s)
• This large stripe is then written sequentially to free space on on the NL-SAS disks, far more efficiently than writing random small block data to the same type of disks.
• The stripe is analysed by Nimbles caching algorithm and if required is also written sequentially to the SSD’s for future random read cache purposes.

For more in-depth look at the tech. Nimble has plenty of resources here:
http://www.nimblestorage.com/resources/datasheets.php
http://www.nimblestorage.com/resources/videos-demos.php

In summary, Nimble uses the NL-SAS for large block sequential writes, SSD for random read cache and compresses data in-line at no performance loss.

To get a solid idea of how the Nimble performed, I fired up a Windows 2008 R2 VM (2 vCPU/4GB RAM) on vSphere 5.1 backed by a 200GB thin provisioned volume from the Nimble over 10GbE iSCSI.

Using the familiar IOMeter, I did a couple of quick benchmarks:

All done with one worker process, tests run for 5 minutes.

80/20 Write/Read, 75% Random, 4K Workload using a 16GB test file (32000000 sectors) and 32 outstanding I/Os per target

24,600 IOps, averaging 1.3 m/s latency, and 95MB/s

50/50 Write/Read, 50% Random, 8K Workload using a 16GB test file (32000000 sectors) and 32 outstanding I/Os per target

19,434 IOps, averaging 1.6 m/s latency and 152MB/s

50/50 Write/Read, 50% Random, 32K Workload using a 16GB test file (32000000 sectors) and 32 outstanding I/Os per target

9,370 IOps, averaging  3.6 m/s latency and 289MB/s

Initial thought is – Impressive throughput while keeping latency incredibly low

That’s all I have had time to play with at this point, if anyone has any suggestions for benchmarks or other questions, I am quite happy to take a look.

Cleaning up “Already used” desktops in VMware View

The “Already used” error is a common problem that exists with using floating (non-persistent) desktops in a completely stateless way (i.e you have assigned the pool desktops to refresh on log-off)

However, a user can (in some circumstances) shutdown or reset their floating desktop and this does not give the View Management server a chance to refresh the desktop back to its “known good” internal disk ready for another user.

This leaves the desktop in an error state, which required manual admin intervention or scheduled scripts to handle the issue.

Of course you should be using AD group policy, you can then remove the ability of the user to shutdown the PCoIP windows desktop leaving them with just the ability to log-off.

But in reality I have still seen the “Already used” error crop up due to administrators haphazardly and incorrectly resetting floating desktops and power or individual server issues causing pools on local disk to come up in this state.

With View 5.1.2, VMware have added a an LDAP attribute called pae-DirtyVMPolicy

This is a per pool setting that allows the administrator to control how “Already used” desktops are treated.

There are three policy settings:

pae-DirtyVMPolicy=0 – This is the default behavior of leaving the desktop in the error state and not available for use.

pae-DirtyVMPolicy=1 – This allows desktops that were not cleanly logged off to be available without being refreshed. The desktop is available in the pool for another user.

pae-DirtyVMPolicy=2 – This setting will automatically refresh a desktop in the “already used” state and make it available again in the pool.

To change the setting on a pool, fire up ADSI Edit on a View Connection server and connect to dc=vdi, dc=vmware, dc=int (see screenshot)

Browse down to Server Groups and you will see your desktop pools on the right pane

Double click a pool name to edit the attributes

Find the LDAP attribute pae-DirtyVMPolicy in the list and change its value from <not set> to 1 or 2 (depending on your requirement)

Hopefully this can help ease some of the administrative burden of large floating desktop pools and the small issues they can have.

For more information on VMware View 5.1.2, see the release notes here:

http://www.vmware.com/support/view51/doc/view-512-release-notes.html

Change the IP address of VMware vCloud Director 5.1 virtual appliance

This blog post is mostly for my own lab documentation, but hopefully someone else may find it useful.

The vCloud Director appliance is built upon 64-bit SuSE Enteprise linux with an embedded version of Oracle 11g Express, which means its setup and troubleshooting are a little different from the supported Redhat Enterprise and Oracle platform.

If you had set up your vCD appliance initially with DHCP on both primary and console proxy interfaces, you may run into issues changing it later, as there are few places to look.

The first thing is to SSH into the vCD appliance and logon as root (you will have to enable root logon=yes in /etc/ssh/sshd_config and restart the sshd daemon first)

Stop the vCloud services; service vmware-vcd stop

cd to $VCLOUD_HOME/etc (which should be /opt/vmware/vcloud-director/etc)

Use vi or vim to edit the global.properties file and change the following IP addresses to what you need

vcloud.cell.ip.primary
consoleproxy.host.https
vcloud.cell.ips

Out of the box the vCD appliance does not have the correct Oracle variables set.

So the following commands are necessary:

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/xe

export PATH=$ORACLE_HOME/bin:$PATH

export ORACLE_SID=XE

You should then be able to successfully run sqlplus with the default username and password for the vCloud DB of vcloud/VCloud.

sqlplus vcloud/VCloud

At the SQL> prompt type:

select name from cells;

This will tell you the name of the director cell you are on

Then type: (substituting the appropriate IP address and cell name details)

update cells set primary_ip=’your-primary-ip’ where name=’name-of-directorcell’;

It should say 1 row updated

Exit from sqlplus

Restart the vCloud services; service vmware-vcd start

You should now be able to access the vCloud admin site on the new primary IP