What makes up a Nimble Storage array?

I have been asked a couple of times about what is inside a Nimble Storage array in terms of hardware.

People have assumed that there may have been proprietary components in a high performance array, but Nimble’s architecture is completely leveraged off a commodity chassis, off the shelf drives and multi-core Intel CPU’s.

Most of the hardware is best illustrated with some photos:

First up – the chassis. Nimble uses a SuperMicro unit with hot swap controllers, power supplies and fans: http://www.supermicro.com/products/nfo/sbb.cfm

WP_20130511_009

This is the front view of the chassis (in this case a Nimble CS260G). It has twelve 3TB Seagate Enterprise NL-SAS drives (model: Constellation ES.2)
http://www.seagate.com/internal-hard-drives/enterprise-hard-drives/hdd/enterprise-capacity-3-5-hdd/

It also has four 300GB Intel 320 SSD’s for read cache purposes.

WP_20130511_010The rear of the chassis, showing the redundant power modules and controllers (with PCIe 10GbE adapters) and dual 1GbE NICs.

Looking at the controller itself:

WP_20130511_004

It’s a Supermicro board (X8DTS-F), with 12GB Kingston ECC DDR3 memory and with what appears to be an Intel 55xx Nehalem family CPU (four core/8 threads). On the far back side are the PCIe slots. Far left is an LSI SAS adapter.

WP_20130511_006

In the top PCIe slot, sits the NVRAM card (for fast write caching on inbound I/O). It appears to be an NVvault 1GB DDR2 module, that is built by Netlist:
http://www.netlist.com/products/vault-data-protection/nvvault-ddr2/

WP_20130511_007In the bottom PCIe slot sits the 10GbE dual port network adapter, which from the model number would appear to be the PE210G2SPi9 component from Silicom:
http://www.silicom-usa.com/Networking_Adapters/PE210G2SPi9-Dual_Port_10_Gigabit_Ethernet_PCI_Express_Server_Adapter_Intel_based_49

That about wraps up the commodity components inside a Nimble array.

What is really apparent, it that cost effective, reliable, high performance/capacity commodity disks, multi-core CPU’s, standard chassis have enabled array vendors like Nimble (and others) to concentrate on engineering their software architecture around data services, file systems, user interfaces and seamless array metrics collection enabling support and data mining opportunities.

Untitled

Renaming vSphere Datastore device display names

In vCenter, storage devices or volumes/LUNs present themselves with a unique identifier usually prefixed with eui. or naa.

These are known as Network Address Authority Identifiers and Extended Unique Identifiers. (part of FC and iSCSI naming format standards for LUNs)

1a

However, when looking at them in vSphere and via the esxcli, these unique identifiers are less than useful for identifying which VMFS datastores you are dealing with when looking at paths selection etc.

For that reason, I usually rename each of these via the vSphere client as I add them.

1

This also benefits when filtering NMP devices from the esxcli

If you have connected to the command line of an ESXi host, you can grep for your renamed display name on a storage device

2

esxcli storage nmp device list | grep -A 9 VMFSProd

3You can see that it picks up the Device Display Name set via the vSphere client as well as returning the 9 rows of available information about that NMP device

Hyper-V converged network configuration with two NICs

Just a script I put together for my Server 2012 Hyper-V hosts that only have two 10Gbe physical ports, where I want to separate various network functions of Hyper-V into separate virtual network adapters.

It also lets me use MPIO with two virtual network adapters dedicated for iSCSI.

This is useful for iSCSI storage vendors who recommend using Windows MPIO with their arrays, e.g HP StoreVirtual and Nimble Storage.

With older versions of Hyper-V (Windows 2008 R2), vendor NIC teaming and iSCSI was not recommended, so you were left with dedicating two physical NICs just for an iSCSI MPIO configuration. This would normally mean you would need at least 4 NICs in your host. (iSCSI and other Hyper-V virtual switch functions)


#Create new NIC team using two existing physical adapters in host, set LB and teaming mode
New-NetLbfoTeam -Name NICTeam1 -TeamMembers pNIC1,pNIC2 -LoadBalancingAlgorithm TransportPorts -TeamingMode SwitchIndependent

#Create new virtual switch using the NIC team created prior. Set to not allow hyper-v mgmt
New-VMSwitch -Name ConvergedHyperSwitch -NetAdapterName NICTeam01 -AllowManagementOS $False -MinimumBandwidthMode Weight

#Create five virtual adapters for various components (LM,mgmt,cluster,iSCSI)
Add-VMNetworkAdapter -ManagementOS -Name "Management" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "LiveMigration" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "CSV" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "iSCSI01" -SwitchName "ConvergedHyperSwitch"
Add-VMNetworkAdapter -ManagementOS -Name "iSCSI02" -SwitchName "ConvergedHyperSwitch"

#If required, set VLAN access for your virtual adapters
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "Management" -Access -VlanId 100
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "CSV" -Access -VlanId 100
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "LiveMigration" -Access -VlanId 100
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "iSCSI01" -Access -VlanId 140
Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName "iSCSI02" -Access -VlanId 140

#Set minimum bandwidth in weighting (0-100)
Set-VMNetworkAdapter -ManagementOS -Name "LiveMigration" -MinimumBandwidthWeight 20
Set-VMNetworkAdapter -ManagementOS -Name "CSV" -MinimumBandwidthWeight 10
Set-VMNetworkAdapter -ManagementOS -Name "Management" -MinimumBandwidthWeight 10
Set-VMNetworkAdapter -ManagementOS -Name "iSCSI01" -MinimumBandwidthWeight 20
Set-VMNetworkAdapter -ManagementOS -Name "iSCSI02" -MinimumBandwidthWeight 20

#Set IP addresses/subnet on interfaces 
New-NetIPAddress -InterfaceAlias "vEthernet (Management)" -IPAddress 10.201.201.201 -PrefixLength "16" -DefaultGateway 10.201.1.1 
Set-DnsClientServerAddress -InterfaceAlias "vEthernet (Management)" -ServerAddresses 10.200.3.21, 10.200.3.96 
New-NetIPAddress -InterfaceAlias "vEthernet (LiveMigration)" -IPAddress 10.201.201.202 -PrefixLength "16" 
New-NetIPAddress -InterfaceAlias "vEthernet (CSV)" -IPAddress 10.201.201.203 -PrefixLength "16" 
New-NetIPAddress -InterfaceAlias "vEthernet (iSCSI01)" -IPAddress 10.190.201.120 -PrefixLength "24" 
New-NetIPAddress -InterfaceAlias "vEthernet (iSCSI02)" -IPAddress 10.190.201.121 -PrefixLength "24" 

Optimising vSphere path selection for Nimble Storage

THIS POST IS NOW OBSOLETE. Use Nimble Connection Manager instead

This post is mostly for documentation or a quick how-to for changing the default path selection policy (PSP) so it is more suitable for Nimble arrays.

For Nimble Storage by default, ESXi 5.1 uses the ALUA SATP with a PSP of most recently used (MRU)

Nimble recommends that this be altered to round robin.

From the CLI of each ESXi host (or from the vMA):

esxcli storage nmp satp set -s VMW_SATP_ALUA -P VMW_PSP_RR

After this point any volume presented from the Nimble array and configured as a VMFS datastore will appear with round robin load balancing and I/O active on all paths.

psp

If you had previously added volumes before changing the default PSP, you can change all existing Nimble volumes to round robin with PowerCLI

#Get all hosts and store in a variable I have called $esxihosts

 $esxihosts = get-vmhost

#Gather all SCSI luns presented to hosts where the vendor string in the LUN description equals “Nimble” and set the multi-pathing policy

Get-ScsiLun -VmHost $esxihosts | where {$_.Vendor -eq "Nimble"} | Set-ScsiLun -MultipathPolicy "roundrobin"

Nimble also make a recommendation around altering the IOps per path setting for round robin. This by default is 1000 IOps before changing data paths.

The official recommendation it to set it to zero (0) for each volume on each host.

Nimble support confirms that this ignores IOps per path and bases path selection on queue depth instead.

So from esxcli on each host, do the following which will search for the string Nimble on all storage devices presented to the host and then loop through and set IOps to zero

for i in `esxcli storage nmp device list | grep Nimble` ; do esxcli storage nmp psp roundrobin deviceconfig set --type "iops" --iops=0 --device=$i; done

I haven’t tested or implemented this as of yet.

I believe Nimbles internal support community has a single powercli script to cover everything here.

It would not be much work to take the per host esxcli stuff into powercli with get-esxcli and aggregate it all together.

Storage Efficiency with Nimble Compression

In my last post I talked about performance of the Nimble Storage platform I was implementing.

In a contended virtualised environment, storage is often the performance sore point or Achilles heal due to insufficient IOps and poor latency of the storage design.

The other major consideration is how efficient the storage platform is at managing the available capacity and what can be done to “do more with less”

With the Nimble platform I am using – this problem is addressed by using:

  • In-line compression on all workloads

Nimble uses a variable block compression algorithm they say can yield a 30-70% saving without altering performance or latency.

  • Thin provisioning

Ubiquitous across most array vendors, variations in block size can result in differences in efficiency.

  • VAAI UNMAP for space reclaimation

Nimble supports the VAAI primitive UNMAP for block reclaimation. This is done from the esxcli of a vSphere host as per this VMware KB:

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014849

For testing I’ll concentrate on the compression and how it fares for a SQL server in a vSphere 5.1 environment.

I took a test SQL server running Windows 2008/SQL 2008 that had disk utilisation around 88%  (130GB of data on 149GB of available disk)

sqldisks

I then performed a storage vMotion migration from the original VMFS datastore on an HP LeftHand volume to a 200GB thin provisioned VMFS datastore presented from the Nimble array.

The result was a 2.25 x saving in space (compressed from 130GB to 56GB)

volcompress

Other VMs such as file/print, remote desktop hosts, infrastructure VMs in the test environment have anywhere between 1.7x to 2.0x compression levels.

In addition to the savings on primary volumes, compression is also applied to snapshots.

Who can say no to close to double the effective capacity on all virtualised workloads with no extra datacenter footprint?

Initial Performance Impressions of Nimble Storage

Recently I added a Nimble CS240G to my vSphere production storage environment.

IMG_1406

The Nimble is affordable and packs some serious features and performance in a 3u enclosure.

The CS240G has 24TB of raw capacity, comprised of twelve 2TB 7200rpm Western Digital enterprise RE4 SATA drives and four Intel 160GB SSDs (Intel 320)

IMG_1404

It has redundant controllers, one active, one on standby (mirrored). There are dual 10GbE ports on each controller as well as dual 1GbE.

You can choose to have the data and management sharing one network or split it out. In my case I went with a redundant management setup on the two 1GbE ports on one VLAN and iSCSI data on the two 10GbE ports on another VLAN.

Nimble does smart things around its ingestion of data. The general process is:

  • Inbound writes, cached in NVRAM and mirrored to the standby controller, write acknowledgement is given to the host at this point (latency measured in micro seconds)
  • The write IO’s are compressed inline in memory and a 4.5MB stripe is built (from possibly many thousands of small IO’s)
  • This large stripe is then written sequentially to free space on on the NL-SAS disks with only an impact to the array of around 11 IOps. Far more efficient than writing random small block data to the same type of disks.
  • The stripe is analysed by Nimbles caching algorithm and if required is also written sequentially to the SSD’s for future random read cache purposes.

For more in-depth look at the tech. Nimble has plenty of resources here:
http://www.nimblestorage.com/resources/datasheets.php
http://www.nimblestorage.com/resources/videos-demos.php

In summary, Nimble uses the NL-SAS for large block sequential writes, SSD for random read cache and compresses data in-line at no performance loss.

To get a solid idea of how the Nimble performed, I fired up a Windows 2008 R2 VM (2 vCPU/4GB RAM) on vSphere 5.1 backed by a 200GB thin provisioned volume from the Nimble over 10GbE iSCSI.

Using the familiar IOMeter, I did a couple of quick benchmarks:

All done with one worker process, tests run for 5 minutes.

80/20 Write/Read, 75% Random, 4K Workload using a 16GB test file (32000000 sectors) and 32 outstanding I/Os per target

24,600 IOps, averaging 1.3 m/s latency, and 95MB/s

50/50 Write/Read, 50% Random, 8K Workload using a 16GB test file (32000000 sectors) and 32 outstanding I/Os per target

19,434 IOps, averaging 1.6 m/s latency and 152MB/s

50/50 Write/Read, 50% Random, 32K Workload using a 16GB test file (32000000 sectors) and 32 outstanding I/Os per target

9,370 IOps, averaging  3.6 m/s latency and 289MB/s

Initial thought is – Impressive throughput while keeping latency incredibly low

That’s all I have had time to play with at this point, if anyone has any suggestions for benchmarks or other questions, I am quite happy to take a look.

Cleaning up “Already used” desktops in VMware View

The “Already used” error is a common problem that exists with using floating (non-persistent) desktops in a completely stateless way (i.e you have assigned the pool desktops to refresh on log-off)

However, a user can (in some circumstances) shutdown or reset their floating desktop and this does not give the View Management server a chance to refresh the desktop back to its “known good” internal disk ready for another user.

This leaves the desktop in an error state, which required manual admin intervention or scheduled scripts to handle the issue.

0a

Of course you should be using AD group policy, you can then remove the ability of the user to shutdown the PCoIP windows desktop leaving them with just the ability to log-off.

But in reality I have still seen the “Already used” error crop up due to administrators haphazardly and incorrectly resetting floating desktops and power or individual server issues causing pools on local disk to come up in this state.

With View 5.1.2, VMware have added a an LDAP attribute called pae-DirtyVMPolicy

This is a per pool setting that allows the administrator to control how “Already used” desktops are treated.

There are three policy settings:

pae-DirtyVMPolicy=0 – This is the default behavior of leaving the desktop in the error state and not available for use.

pae-DirtyVMPolicy=1 – This allows desktops that were not cleanly logged off to be available without being refreshed. The desktop is available in the pool for another user.

pae-DirtyVMPolicy=2 – This setting will automatically refresh a desktop in the “already used” state and make it available again in the pool.

To change the setting on a pool, fire up ADSI Edit on a View Connection server and connect to dc=vdi, dc=vmware, dc=int (see screenshot)

1

Browse down to Server Groups and you will see your desktop pools on the right pane

2

Double click a pool name to edit the attributes

3

Find the LDAP attribute pae-DirtyVMPolicy in the list and change its value from <not set> to 1 or 2 (depending on your requirement)

Hopefully this can help ease some of the administrative burden of large floating desktop pools and the small issues they can have.

For more information on VMware View 5.1.2, see the release notes here:

http://www.vmware.com/support/view51/doc/view-512-release-notes.html