IBM BladeCenter Virtual Fabric solution and vSphere

A basic of what’s required to get the IBM BladeCenter Virtual Fabric solution running with VMware vSphere 4.0

As of late 2009, IBM offer a virtual I/O solution in their BladeCenters to compete with HP and their Virtual Connect Flex-10 technology.

This combination consists of Emulex Virtual Fabric adapters (VFA) in each blade and BNT Virtual Fabric switch modules in the BladeCenter chassis.

The Emulex VFA ships as a CFFh card for the blade servers and the BNT Virtual Fabric module is designed for the high speed bays (7 and 9) of an H/HT blade chassis from IBM.

The setup to get this working:

BladeCenter H with HS22/HS22V blades with latest firmware
Emulex Virtual Fabric Adapters with latest 2.101.411.7 firmware and bootcode (the only firmware supported by IBM at this time to work correctly with the Bladecenter/Blades)
The 2.101.386.0 driver for ESX 4.0 (the only vSphere driver available for the Emulex VFA) (See updated driver info at the end of this post – Feb 2011)

All firmware and drivers from the Emulex site here:
http://www.emulex.com/downloads/ibm/oneconnect-software-kits/ocm10102-n-x-vmware-sw-kit.html

For the BNT switches – 6.1.3 firmware is the latest as of writing.
You will need to register your BNT modules to get the firmware from here: http://downloads.bladenetwork.net

By default the virtual fabric adapters are set to vNIC mode – this means eight vNICs (four per internal port on the card)

If you wish, you can change the mode to pNIC via the blade servers uEFI firmware setup -you will then have the traditional two 10Gbe ports available to your host OS.

When the Emulex VFA’s talk to the BNT Virtual Fabric modules, all eight vNICs are set at 2.5Gbps by default – evenly divided between the two 10Gbps physical ports on the card.

The speed of the vNIC is controlled via the BNT switch interface: (it can be fine tuned from 100Mbps to 10Gbps)

At this point in time, ESX/ESXi 4.0 Update 1 does not come bundled with the driver for the Emulex VFA, so the device appears as an unknown ethernet controller to ESX: (output using lspci)

Downloading the driver iso via the Emulex site or from the VMware site here: http://downloads.vmware.com/d/details/esx_esxi40_serverengines_dt/ZHcqYmQlaEBiZGVqdA==

After installing and rebooting, we now have eight vNICS (The Emulex VFA is based on the ServerEngine BladeEngine 2 chip)

The enabled vNICs appear in the vSphere client like this: (you can have as few as two or as many as eight enabled if you wish, depending on external uplinks from the BNT switch)

vmnic2,4,6,8 are vNICS from internal port 0 of the EmulexVFA – which maps to the BNT Virtual Fabric switch in Bay7 of the BladeCenter

vmnic3,5,7,9 are vNICS from internal port 1 of the Emulex VFA- which maps to the BNT Virtual Fabric switch in Bay9 of the BladeCenter

An update (Feb 2011) – Emulex continue to release updated bladeengine/serverengine drivers at a furious pace for ESX(i).

This is due to the numerous problems these drivers have had (many HP and IBM customers will know )

These problems include NMI errors , PSOD (purple screens of death in ESX), packet loss, VLAN and interface redundancy strangeness.

The latest sets of drivers are here. (ver. 2.103.377.0 – dated 10 Feb 2011)

http://downloads.vmware.com/d/details/esxi4x_emulex_be10gb_dt/ZCV0YnRkKmpidHdw

It’s claimed on other sites and by HP that most issues are fixed.

They do seem to fix the PSOD and ping issues I was having with the Emulex VFA in my HS22V blades under ESXi 4.0 and 4.1

About these ads
This entry was posted in Blade Servers, IBM, VMware vSphere and tagged , , , , , , , , , , , . Bookmark the permalink.

7 Responses to IBM BladeCenter Virtual Fabric solution and vSphere

  1. fluke says:

    Great post! I am in the midst of configuring my Virtual Fabric and is having problems with the uplink and switch group. Cannot wait for your next post!!

  2. jbrison says:

    Hello,
    Do you have issue with your configuration?
    When CDCEther (Eth over Eth for IMM) is enable, I get lots of problems like don’t see the change of bandwith or some vNIC are not connected.
    And something after a reboot a get no connection to network even if I see the port Up.

    • Barrie says:

      There is a bug with the current Emulex VFA 2.101.386.0 driver for ESX 4.0 that means that vNIC speed changes are not shown (in fact it shows 0 as the speed)

      I was told that new drivers are not far away (July – perhaps in conjunction with vSphere/ESX 4.1)

      There is also new firmware (late May) for the BNT Virtual Fabric switches (6.3) which adds and fixes a few things (see IBM BladeCenter support site for this)

      Other than that – no other issues

      I prfer to run the VFAs as two 10GbE pnics with ESX portgroups and VLANs to seperate stuff these days – simplistic and manageable

      Hope this helps somewhat

  3. jbrison says:

    thanks for the reply, I think it will help.
    Unfortunately, I only have to wait for the new firmware.

  4. jarl-dv says:

    Hi Barrie!

    I was looking for websites helping me set up exactly what you describe in this post. And then when things were getting interesting, you stop the blog post.

    My question is this: When do you plan on posting the next step in this article?

    Thank you in advance! And thanks for a great post.

    Regards,
    -Jarle-

    • Barrie says:

      Probably not going to re-visit this.
      I use the Emluex VFA cards in pNIC mode – i.e. 2 x 10Gbe in each blade – just simpler and the Emulex vNic driver side of things I found a tad flaky under ESX 4/4.1 under certain conditions

      The BNT PDFs on their site (Application guides I think) give you all you need to know about groups and uplinks

  5. Yann Bizeul says:

    I think there is one catch here : how about using Embedded ESXi ? My understanding is you’re busted, still no support in standard dd image nor IBM provided (and crappy, mot of the time) thumb sticks.

    I posted some kind of solution here http://communities.vmware.com/thread/269856, but if you guys have better ideas, please suggest !

Comments are closed.