A basic of what’s required to get the IBM BladeCenter Virtual Fabric solution running with VMware vSphere 4.0
As of late 2009, IBM offer a virtual I/O solution in their BladeCenters to compete with HP and their Virtual Connect Flex-10 technology.
The Emulex VFA ships as a CFFh card for the blade servers and the BNT Virtual Fabric module is designed for the high speed bays (7 and 9) of an H/HT blade chassis from IBM.
The setup to get this working:
BladeCenter H with HS22/HS22V blades with latest firmware
Emulex Virtual Fabric Adapters with latest 2.101.411.7 firmware and bootcode (the only firmware supported by IBM at this time to work correctly with the Bladecenter/Blades)
The 2.101.386.0 driver for ESX 4.0 (the only vSphere driver available for the Emulex VFA) (See updated driver info at the end of this post – Feb 2011)
All firmware and drivers from the Emulex site here:
For the BNT switches – 6.1.3 firmware is the latest as of writing.
You will need to register your BNT modules to get the firmware from here: http://downloads.bladenetwork.net
By default the virtual fabric adapters are set to vNIC mode – this means eight vNICs (four per internal port on the card)
If you wish, you can change the mode to pNIC via the blade servers uEFI firmware setup -you will then have the traditional two 10Gbe ports available to your host OS.
When the Emulex VFA’s talk to the BNT Virtual Fabric modules, all eight vNICs are set at 2.5Gbps by default – evenly divided between the two 10Gbps physical ports on the card.
The speed of the vNIC is controlled via the BNT switch interface: (it can be fine tuned from 100Mbps to 10Gbps)
At this point in time, ESX/ESXi 4.0 Update 1 does not come bundled with the driver for the Emulex VFA, so the device appears as an unknown ethernet controller to ESX: (output using lspci)
Downloading the driver iso via the Emulex site or from the VMware site here: http://downloads.vmware.com/d/details/esx_esxi40_serverengines_dt/ZHcqYmQlaEBiZGVqdA==
After installing and rebooting, we now have eight vNICS (The Emulex VFA is based on the ServerEngine BladeEngine 2 chip)
The enabled vNICs appear in the vSphere client like this: (you can have as few as two or as many as eight enabled if you wish, depending on external uplinks from the BNT switch)
vmnic2,4,6,8 are vNICS from internal port 0 of the EmulexVFA – which maps to the BNT Virtual Fabric switch in Bay7 of the BladeCenter
vmnic3,5,7,9 are vNICS from internal port 1 of the Emulex VFA- which maps to the BNT Virtual Fabric switch in Bay9 of the BladeCenter
An update (Feb 2011) – Emulex continue to release updated bladeengine/serverengine drivers at a furious pace for ESX(i).
This is due to the numerous problems these drivers have had (many HP and IBM customers will know )
These problems include NMI errors , PSOD (purple screens of death in ESX), packet loss, VLAN and interface redundancy strangeness.
The latest sets of drivers are here. (ver. 2.103.377.0 – dated 10 Feb 2011)
It’s claimed on other sites and by HP that most issues are fixed.
They do seem to fix the PSOD and ping issues I was having with the Emulex VFA in my HS22V blades under ESXi 4.0 and 4.1